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Site  Integration  Through  Hierarchical  Control 


R.  Lumia,  Robot  Systems  Division,  NIST 
A.  Sanderson,  RPI 
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1.  Goal 

The  goal  of  this  work  is  to  reduce  construction  time  by  half  through  the  use  of  hierarchical  con- 
trol for  the  integration  of  all  activities  at  the  worksite.  It  will  accomplish  this  through: 

1)  Automating  planning  and  scheduling  directly  from  databases  of  project  planners,  design  en- 
gineers, contractors,  and  materials  suppliers. 

2)  Generating  detailed  plans  directly  from  automated  survey  data. 

3)  Increasing  the  efficiency  with  which  men  and  machines  are  used  by  ensuring  that  all  neces- 
sary tools,  materials,  and  personnel  are  available  at  the  proper  places  and  times. 

4)  Optimally  rescheduling  use  of  men  and  machines  in  real-time  in  response  to  short-term  dis- 
ruptions caused  by  weather,  equipment  malfunction  and  similar  unpredictable  events. 

5)  Eliminating  time  spent  searching  for  materials  by  automatically  tracking  and  controlling  ma- 
terials delivery  and  storage. 

6)  Increasing  the  efficiency  of  machinery  operators  by  providing  them  with  exact  visual  guid- 
ance for  earth  removal,  materials  placement,  and  site-positioning. 

7)  Reducing  re-work  by  enhancing  the  accuracy  of  siting,  placing,  cutting,  and  forming  opera- 
tions, controlling  materials  flows,  and  by  directly  guiding  work  from  master  plans  and  designs 
in  real-time. 

8)  Acquiring  as-built  information  as  the  work  occurs  to  reduce  turn-around  time  for  checking 
and  correcting  operations. 

In  addition  to  reducing  construction  time,  the  proposed  work  will  reduce  materials  costs  by  reduc- 
ing waste,  and  enhance  safety  by  monitoring  equipment  in  motion. 

2.  Background 

2.1.  Root  of  the  problem 

It  is  well  known  that  unforeseen  events  can  idle  equipment  at  the  worksite,  which  wastes  time 
and  costs  money.  A traffic  jam  can  result  in  the  rejection  of  a cement  delivery.  Overnight  rain  can 
fill  a hole  and  work  cannot  proceed  until  a pump  arrives.  An  event  can  occur  which  requires  equip- 
ment not  at  the  worksite.  Normally,  it  is  the  job  of  the  site  foreman  to  deal  with  these  events.  Our 
argument,  however,  is  that  the  foreman  can  only  oversee  his  “island”  of  activity  and  therefore  is 
implementing  a local  rather  than  global  optimization  of  a construction  plan.  The  foreman’s  funda- 
mental limitation  is  that  he  lacks  information  about  what  is  happening  at  the  other  worksites  of  the 
project.  Consequently,  the  key  problems  are  the  lack  of  global  information  and  a method  for  using 
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it  effectively. 

A site  foreman  does  an  excellent  job  in  allocating  resources  at  his  worksite,  i.e.,  he  does  the 
best  job  he  can  with  the  information  he  has.  However,  this  local  optimization  results  in  sub-optimal 
use  of  equipment,  sub-optimal  planning  for  materials  delivery,  sub-optimal  use  of  personnel.  The- 
oretically, if  the  foreman  knew  about  the  state  of  all  of  the  project  worksites,  resources  could  be 
allocated  more  effectively.  However,  in  practice,  the  foreman  would  drown  in  the  details.  He  needs 
tools  to  help  him  sort  out  the  complexity  in  order  to  enhance  his  performance.  Hierarchically  orga- 
nized control  systems  address  this  issue  of  complexity  and  provide  a tool  to  make  construction  sites 
more  efficient. 

2.2.  Advantages  and  applications  of  hierarchical  control 

Hierarchical  control  is  not  a new  idea.  It  was  practiced  in  China  around  3000  B.C.  It  is  com- 
monly found  in  the  military,  and  most  business  and  government  organizations.  Recently,  these  con- 
cepts have  been  applied  to  controlling  machines.  Hierarchical  control  has  been  used  for  many  sys- 
tems including 

• Factory  control  in  an  automated  manufacturing  research  facility  at  NIST 

• Factory  workstation  control 

• Machine  tool  loading  and  unloading 

• Robotic  debarring 

• Advanced  controller  for  machine  tools,  robots,  and  coordinate  measuring  machines 

• Advanced  inspection  system 

• Vision-guided  robot  manipulation 

• Vision-based  automated  highway  driving 

• Unmanned  land  and  undersea  vehicles 

• Submarine  operation  automation 

• Coal  mine  automation 

• Space  Station  telerobotic  servicer 

• Robot  crane 

While  this  set  of  applications  may  appear  disparate,  the  common  thread  is  that  complex  systems 
require  coordinated  activities  in  order  to  achieve  optimal  performance.  We  argue  that  construction 
sites  are  also  complex  systems,  and  therefore  are  amenable  to  hierarchical  control  techniques. 

3.  Hierarchical  Control  for  Site  Integration 

3.1.  Hierarchical  control  overview 

A hierarchical  control  architecture  can  integrate  many  people  and  machines  into  a coordinated 
system.  It  forms  the  “glue”  which  connects  measurement  technologies  and  equipment  in  order  to 
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enhance  performance  while  reducing  time  and  waste.  A good  hierarchical  control  system  should 
provide  an  engineering  methodology  for  the  design  and  implementation  of  the  system.  It  should 
define  how  global  plans  can  be  decomposed  into  local  actions  that  can  be  executed  by  production 
equipment.  Also,  it  should  allow  any  amount  of  automation,  i.e.,  it  should  support  the  entire  range 
from  teleoperation  to  autonomy,  in  its  attempt  to  balance  cost  and  performance.  This  is  possible 
because  hierarchical  control  systems  deal  with  planning  horizons  ranging  from  years  and  months, 
down  to  jobs  lasting  weeks,  days,  and  hours,  as  well  as  for  tasks  lasting  minutes,  seconds,  and  even 
milliseconds. 

Some  of  the  elements  of  a hierarchical  control  system  architecture  and  how  they  pertain  to  site 
integration  are  shown  in  Figure  1.  There  are  three  parallel  hierarchies:  task  decomposition,  world 
modeling,  and  sensory  processing.  Task  decomposition,  as  its  name  suggests,  decomposes  a task 
into  smaller  and  smaller  subtasks,  eventually  culminating  in  commands  sent  to  the  equipment.  The 
world  model  stores  a representation  of  the  worksite.  Sensory  processing  collects  and  analyzes  data 
to  keep  the  models  in  the  world  model  in  registration  with  reality. 

Figure  1 also  introduces  the  concept  of  multiple  levels  of  control,  only  two  of  which  are  shown 
in  the  figure.  Each  level’s  planning  horizon  is  roughly  10  times  longer  than  the  level  below  it.  For 
example,  the  task  decomposition  box  performing  the  strategic  long-term  plan  could  be  planning  a 
day’s  worth  of  activities  while  the  level  below  is  concerned  with  a sequence  of  tasks  which  require 
15  minutes  each.  At  the  lowest  level,  the  world  model  maintains  an  up  to  date  model  of  the  site, 
e.g.,  the  site  model  could  be  concerned  with  the  functionality  of  the  equipment,  precisely  where 
each  piece  of  equipment  is  and  what  it  is  doing,  and  the  current  state  of  the  task  executing.  At  a 
higher  level,  where  the  time  frames  are  more  broad,  the  world  model  could  store  the  CAD  model 
of  the  road,  the  desired  end  product,  as  well  as  the  as-built  model,  the  reality  of  the  constructed 
road.  Similarly,  sensor  processing  tasks  could  be  divided  between  short  time  frames,  e.g.,  gathering 
specific  road  measurements  for  a small  part  of  the  road,  and  longer  time  frames,  e.g.,  integrating 
those  short  term  measurements  into  a global  measurement  for  longer  road  segments. 

In  applying  hierarchical  control  to  various  applications,  there  is  significant  flexibility  in  choos- 
ing appropriate  sensors,  planners,  etc.,  and  configuring  these  components  into  a real-time  control 
system  to  achieve  any  desired  level  of  automation.  The  same  approach  applies  to  site  integration 
by  implementing  the  system  in  phases  and  demonstrating  the  value  added  for  each  phase. 

Figure  2 shows  how  the  real-time  planner  at  the  lowest  level  of  the  architecture  would  be  used 
to  coordinate  construction  equipment.  Any  desired  level  of  automation  is  possible.  For  example, 
the  system  could  simply  provide  instructions  to  the  human  operators  of  the  equipment.  Alternative- 
ly, it  could  use  the  human  operator  to  control  the  location  of  the  vehicle  but  automatically  control 
the  remainder  of  the  equipment  processes.  The  most  sophisticated  alternative  would  be  to  control 
the  entire  operation  autonomously,  which  may  be  technologically  possible  but  is  probably  too  cost- 
ly at  the  present  time.  In  a similar  fashion.  Figure  3 shows  how  sensors  could  be  connected  and 
configured  into  a real-time  system.  Figures  4,  5,  and  6 show  more  detail  of  a potential  implemen- 
tation of  the  hierarchy  with  different  proportions  of  autonomy. 

3.2.  Stages  of  integration 

Hierarchical  control  allows  technology  to  be  integrated  in  phases.  For  example,  the  output  of 
the  real-time  planner  in  Figure  1,  which  connects  to  equipment,  can  take  different  forms  depending 
on  the  state  of  technology.  In  the  short  term,  the  output  could  be  a screen  with  instructions  to  the 
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foreman  and  equipment  operators.  More  advanced  technology  provides  the  operator  with  the  CAD 
design  superimposed  and  registered  with  the  worksite.  Using  this  graphics  display,  the  operator 
could  place  structural  materials,  forms,  etc.  As  graphics  technology  advances,  virtual  reality  tech- 
niques can  be  incorporated.  Ultimately,  the  commands  could  control  the  equipment  directly  with 
no  human  interaction. 

To  illustrate  the  sensory  side  of  Figure  1,  consider  the  following  scenario.  Truck  65  has  arrived 
with  shipment  16  and  is  supposed  to  dump  its  contents  at  the  worksite.  In  the  short  term,  the  fact 
that  truck  65  has  arrived  with  shipment  16  is  keyed  manually  into  the  system.  The  system  responds 
with  the  instruction  to  dump  contents  at  grid  87.  Then,  the  truck  driver  views  a display  which  in- 
forms him  where  he  is,  so  that  he  can  determine  how  to  reach  grid  87  exactly.  As  technology  ad- 
vances, the  system  could  take  control  of  the  vehicle  once  it  enters  the  worksite,  automatically  ac- 
cess the  database  to  determine  the  desired  location,  drive  the  truck  to  that  place,  and  control  the 
dumping.  In  all  cases,  the  system  would  automatically  update  the  site  model  of  the  material  data- 
base. 

Consider  another  technology.  If  materials  are  bar-coded,  the  materials  database  can  be  updated 
every  time  material  is  moved  from  one  location  to  another.  Inventories  can  be  adjusted,  and  orders 
generated  automatically.  Keeping  track  of  the  location  of  objects  at  the  worksite  goes  beyond  mon- 
itoring materials. 

Position  sensors  can  determine  the  precise  location  of  equipment  during  operations  at  the  work- 
site. For  example,  it  is  possible  to  measure  the  location  of  the  backhoe  bucket  in  real-time.  This 
information  can  be  used  to  update  the  site  database  in  real-time,  which  ultimate  instructs  the  oper- 
ator precisely  where  and  how  to  dig. 

Laser  surveying  instruments  are  commercially  available  to  build  CAD  models  of  the  terrain. 
This  information  needs  to  be  incorporated  into  the  hierarchical  control  system  to  guide  the  crews 
and  equipment  operators.  Ultimately,  when  updated  by  input  from  the  field,  these  data  can  be  used 
to  create  the  as-built  model  of  the  site. 

4.  Scheduling  activities  at  the  worksite 

A road  construction  project  may  be  thought  of  as  a set  of  large-scale  tasks,  such  as  excavation, 
grading,  paving,  bridge  construction,  etc.,  which  must  be  carried  out  in  a special  order  to  accom- 
plish the  final  construction  goal.  The  execution  of  these  tasks  requires  the  careful  coordination  of 
resources  delivered  to  and  removed  from  the  site,  the  availability  of  machines  and  people  at  par- 
ticular places  and  times,  and  the  arrangement  and  transport  of  parts  and  materials  within  the  work- 
site. While  many  different  factors  affect  the  success  of  such  a project,  the  design  itself  and  the  map- 
ping of  that  design  into  a project  plan  are  critical  to  the  efficient  management  and  resulting  quality. 
Computer-based  aids  for  the  development  and  tracking  of  complex  projects  offer  fundamental  ad- 
vantages to  the  road  construction  industry  where  coordination  of  multiple  activities  on  a preset 
schedule  is  vital  to  efficient  and  safe  operation. 

Computer-based  design  tools  are  now  required  by  many  states  for  large  highway  construction 
projects.  In  this  approach,  the  designer  develops  a representation  of  his  highway  project  design  as 
a computer-based  (CAD)  file,  and  can  use  that  file  as  a basis  for  structural  analysis,  materials  se- 
lection, project  costing,  environmental  impact,  and  graphics  rendering  for  presentation.  The  design 
itself  provides  the  common  core,  and  the  computer  representation  enforces  a consistency  among 
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the  various  types  of  analysis.  Such  techniques  are  commercially  available  and  increasingly  used 
for  major  projects.  A CAD-based  design  and  database  representation  will  be  an  essential  compo- 
nent of  the  proposed  demonstration  project. 

While  the  current  CAD  tools  provide  useful  means  to  capture  the  basic  design  of  the  construc- 
tion project,  there  are  many  additional  ways  in  which  this  representation  may  used  for  project  co- 
ordination and  management  Current  CAD  design  methods  are  used  in  the  preliminary  costing  and 
assessment  of  projects,  but  are  not  commonly  used  in  the  planning  and  operations  themselves.  We 
propose  to  bring  the  CAD  design  into  the  project  planning  and  day-to-day  management  of  the 
project  in  order  to  increase  the  efficiency  and  accuracy  of  these  operations.  One  example  of  this 
use  is  the  site  layout  problem.  The  CAD  tools  should  be  developed  to  represent  a sequence  of  con- 
struction phases  depicting  the  spatial  configuration  of  partially  completed  work.  Then,  the  layout 
of  materials  stores,  equipment,  and  personnel  may  be  configured  to  maximize  the  efficiency  of  the 
workplace.  Minimizing  the  transport  and  relocation  of  materials  achieves  a significant  gain  in  op- 
erations efficiency. 

A second  example  of  the  integration  of  CAD  tools  into  road  construction  operations,  is  the  au- 
tomated site  positioning  system.  The  CAD  representation  specifies  the  position,  location,  and 
shape  of  objects  and  surfaces  in  three-dimensions  from  the  design.  For  example,  the  location, 
grade,  and  shaping  of  the  final  road  surface  is  exactly  specified.  However,  in  practice,  it  is  extreme- 
ly difficult  to  map  this  specification  onto  a work  site.  Surveying,  marking,  and  remarking  of  posi- 
tions, elevations,  and  grades  is  a routine  practice  during  road  construction.  Maintaining  accuracy 
throughout  the  many  steps  which  are  required  is  a major  task.  An  automated  site  location  system 
provides  a means  to  locate  exact  positions  and  orientations  of  objects  in  the  three-dimensional 
workspace.  A given  machine,  such  as  a bulldozer,  would  be  tracked  by  beacons  on  the  site,  and  its 
position  and  velocity  would  be  monitored  and  displayed.  Such  systems  are  now  under  experimental 
development  and  demonstration  of  the  integration  of  such  a site  location  system  with  CAD  speci- 
fications would  be  a major  focus  of  the  proposed  demonstration  project. 

Given  a project  design  which  can  be  linked  into  the  operations  of  the  project,  there  are  enor- 
mous opportunities  to  improve  the  task  planning  and  scheduling  for  the  project.  There  are  already 
some  existing  tools  which  are  used  in  road  construction  project  planning.  PERT  (Program  Evalu- 
ation and  Review  Technique)  charts  and  CPM  (Critical  Path  Methods)  are  the  most  common. 
These  methods  provide  a means  to  evaluate  alternative  sequences  of  tasks  within  a complex 
project,  assign  costs  or  penalties  to  different  tasks,  and  choose  which  sequences  are  best  suited  to 
the  given  project.  In  these  approaches,  planning  is  the  process  of  choosing  the  order  of  work  among 
the  tasks  and  is  usually  shown  as  a linked  series  of  nodes  in  the  CPM  diagram.  Scheduling  is  the 
assignment  of  times,  or  time  bounds,  on  these  tasks  and  is  used  to  estimate  the  overall  completion 
time  and  the  critical  times  for  availability  of  resources.  While  these  methods  have  provided  impor- 
tant advances  in  the  systematic  approach  to  project  planning,  they  have  two  principal  deficiencies: 
(1).  lack  of  embedded  knowledge,  and  (2).  lack  of  adaptive  or  dynamic  behavior.  These  two  defi- 
ciencies may  be  addressed  by  additional  new  developments  in  planning  methods. 

PERT  and  CPM  do  not  embed  design  information  or  knowledge  of  the  domain  into  the  plan- 
ning process.  The  formulation  of  the  plan  usually  relies  entirely  on  the  skill  and  experience  of  the 
human  planner  to  assess  and  assimilate  the  many  variable  which  affect  the  planned  sequence  and 
its  variability.  The  development  of  knowledge-based  planning  systems  addresses  this  issue  and  has 
resulted  in  prototype  programs  which  provide  improved  planning  environments  for  specific  do- 
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mains.  In  our  work  on  this  demonstration  project,  we  would  emphasize  the  development  and  dem- 
onstration of  two  specific  systems:  First,  a knowledge-based  site  layout  system  which  would  assist 
in  the  placement  of  materials  and  machines  and  the  optimal  configuration  of  the  site  for  improved 
efficiency  and  safety.  Second,  a knowledge-based  scheduling  system  which  would  optimize  the 
utilization  of  machines  and  personnel  based  on  internal  representation  of  domain  knowledge  about 
the  needs  and  constraints  of  the  road  construction  domain.  Incorporation  of  these  tools  will  have  a 
direct  benefit  to  the  efficiency  of  the  project,  the  utilization  of  equipment  and  facilities,  and  the 
ability  to  monitor  and  assess  progress. 

A PERT  or  CPM  analysis  is  usually  done  prior  to  the  project  and  is  difficult  to  change  or  update 
as  unexpected  events  occur  during  the  course  of  the  project.  This  lack  of  adaptive  or  dynamic  be- 
havior often  renders  the  original  plan  useless  within  a short  period  after  the  project  commences. 
Instead,  new  tools  are  required  which  can  respond  to  perturbations  in  the  constraints  of  the  plan,  or 
dynamically  replan  successive  tasks  as  necessary  during  the  project.  Such  tools  in  effect  become  a 
top  level  of  coordination  for  the  principal  tasks  and  provide  a basis  for  the  hierarchical  decompo- 
sition of  subtasks.  These  discrete-event  modeling  and  control  methods  have  been  used  extensively 
in  the  manufacturing  domain  where  dynamic  changes  in  requirements  and  resources  require  fre- 
quent revision  of  manufacturing  plans.  In  the  case  of  road  construction,  a “discrete-event”  may  be 
thought  of  as  a specific  task  with  well-defined  start  time  and  end  time.  Each  such  task  has  certain 
resources  which  are  required.  For  example,  a trenching  and  pipe-laying  operation  may  require  the 
availability  of  an  excavator,  a crane,  a compactor,  and  the  next  section  of  pipe.  The  availability  of 
these  ‘preconditions’  to  an  event  define  a state  of  the  system  model  and  enable  the  execution  of  the 
event  itself.  The  outcome  of  the  event  is  a new  set  of  state  conditions. 

A powerful  set  of  models  which  represent  these  conditions  and  actions  very  effectively  are 
called  Petri  nets.  The  Petri  net  uses  a circle,  or  ‘node’,  to  represent  the  state  of  a resource,  and  a 
line  with  arrows,  or  ‘transition’,  to  represent  the  execution  of  an  action  (‘firing  of  a transition’). 
Figure  7 illustrates  this  notation  for  Petri  net  models,  and  a sample  Petri  net  model  for  the  trenching 
and  pipelaying  problem  is  shown  in  Figure  8.  In  this  example  the  crane  is  a resource  which  is 
shared  by  two  subtasks,  setting  the  new  pipe  section  and  placing  the  compactor  in  the  trench.  Me- 
diating a shared  resource  in  this  fashion  and  coordinating  parallel  operations  are  particular 
strengths  of  the  Petri  net  representation.  It  is  important  to  recognize  that  this  type  of  model  repre- 
sents a set  of  possible  plans,  rather  than  just  one  plan.  All  of  the  possible  plans  are  consistent  with 
the  constraints  of  the  problem,  but  may  be  initiated  by  different  events.  Thus  the  Petri  net  is  inher- 
ently a dynamic  model  capable  of  responding  to  different  conditions  as  the  arise.  In  addition,  we 
can  attribute  estimated  execution  times  to  different  transitions  and  assess  the  alternative  execution 
time  of  the  whole  task  depending  on  different  possible  conditions  which  may  arise.  We  thus  assess 
the  range  of  possible  outcomes  and  incorporate  that  into  our  planning  process.  Responding  effi- 
ciently to  these  ‘what  if?’  questions  in  the  planning  domain  is  a key  attribute  of  the  Petri  net  mod- 
eling approach,  and  will  provide  a basis  for  coordination  and  control  of  discrete  event  sequences 
during  execution  of  the  project.  Existing  software  tools  are  available  which  can  be  adopted  to  the 
demonstration  project  and  would  provide  a framework  for  modeling  and  planning  at  the  discrete- 
event  level.  As  described  below,  further  decomposition  of  tasks  results  in  a hierarchy  of  continuous 
control  modes  which  can  be  systematically  organized  as  a hierarchical  control  system. 

Implementation  of  the  coordination  and  control  functions  in  a complex  work  environment  such 
as  a construction  site  will  require  attention  to  communications  and  information  transmission  capa- 
bilities. A construction  worksite  may  be  thought  of  as  a network  of  users  and  machines,  each  with 
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different  elements  of  the  overall  task.  However,  the  coordination  of  these  various  functions  will 
require  that  common  elements,  like  the  project  design,  by  available  and  consistent  for  all  of  these 
users.  In  addition,  as  different  aspects  of  the  job  proceed,  it  is  necessary  for  the  status  of  jobs  to 
updated  and  communicated  to  other  parts  of  the  worksite.  Such  a distributed  user  community  con- 
stitutes a network,  and  the  definition  and  implementation  of  networking  principles  for  a highway 
construction  worksite  will  be  a major  component  of  the  proposed  demonstration  project.  This  net- 
work will  require  agreed  upon  modes  of  transmission  on  information  as  well  as  establishment  of 
protocols,  or  standards,  for  relay  of  information.  For  example,  representation  of  the  grade  of  the 
road  in  the  design  database  migru  be  transmitted  to  the  grader  control  system  in  one  form,  but  ac- 
cessed by  the  pavement  compaction  inspection  system  in  another  form.  These  interactions  must  be 
specified  and  enforced  in  order  to  assure  consistency  among  the  different  agents  in  the  network 
system. 

For  the  road  construction  site,  many  of  the  elements  of  the  network  will  be  machines  which  are 
concurrently  carrying  out  real-time  tasks.  This  real-time  distributed  interaction  imposes  still  anoth- 
er requirement  on  the  design  of  the  system.  A standard  for  communications  among  the  real-time 
elements  of  the  system  must  be  adopted,  and  must  take  into  account  the  timing  of  the  individual 
processes  and  their  interactions.  This  requirement  for  a real-time  distributed  operating  system  to 
coordinate  processes  is  common  to  many  other  systems  which  integrate  computers  in  an  interactive 
domain.  One  approach  to  this  problem  utilizes  a message-passing  protocol  which  permits  comput- 
ers to  communicate  through  messages  which  update  the  timing  requirements  and  constraints  from 
each  of  the  other  systems.  The  integration  of  a discrete-event  control  model,  such  as  the  Petri  net, 
with  a real-time  distributed  operating  system  provides  the  framework  for  implementation  of  com- 
munications and  coordination  of  multiple  tasks  in  the  road  construction  demonstration  project  In- 
dividual or  hierarchies  of  continuous  controllers,  sensors,  and  modeling  systems  which  control  the 
motions  of  individual  machines  act  as  nodes  on  this  communications  network  and  report  their  sta- 
tus and  functions  through  the  distributed  operating  system  mechanisms. 

5.  Benefits  of  using  hierarchical  control 

Hierarchical  control  can  help  address  the  root  cause  of  non-optimality  at  worksites:  lack  of  glo- 
bal information  and  an  effective  way  to  use  it.  The  following  table  lists  a particular  capability 
which  can  be  achieved  with  a hierarchical  control  system,  whether  the  time  frame  for  its  implemen- 
tation is  considered  short  term  (1-3  years),  medium  term  (3-5  years),  or  long  term  (5-10  years),  and 
a specific  example  of  that  capability  applied  to  site  integration.  The  benefits  of  using  hierarchical 
control  transcend  the  list,  and  therefore  this  list  should  be  considered  to  be  representative  of  the 
types  of  benefits  one  should  expect. 


Table  1:  Productivity  improvement  through  hierarchical  control 


CAPABILITY 

TIME  FRAME 

EXAMPLE 

Efficient  scheduling 
based  on  variable  work- 
site conditions 

Short 

When  equipment  breaks,  a plan  can  be  gener- 
ated to  optimize  work  for  the  entire  site  by 
transferring  equipment  from  one  work  site  to 
another. 
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Table  1:  Productivity  improvement  through  hierarchical  control 


CAPABILITY 

TIME  FRAME 

EXAMPLE 

Short 

By  having  an  up  to  date  schedule  based  on  the 
actual  state  of  the  sites,  people  are  sent  to  sites 
when  needed  to  perform  specific  tasks,  thereby 
minimizing  idle  time. 

Short 

A weather  report  of  overnight  rain  will  automat- 
ically generate  scheduling  of  pumps  to  arrive  at 
sites  where  holes  were  previously  dug. 

Short 

Material  delivery  schedules  and  equipment  use 
plans  can  be  re-generated  in  real-time  and 
reflect  the  most  up  to  date  information  available 
about  the  worksite. 

Reduce  time  to  locate 
equipment  or  materials 

Short 

When  equipment  breaks,  or  the  state  of  the  site 
changes,  new  equipment  may  be  necessary.  The 
system  can  request  the  “best”  equipment  auto- 
matically by  balancing  the  work  done  at  one 
particular  site  with  the  work  in  progress  at  all 
other  sites. 

Short 

When  new  equipment  is  needed  for  a site,  all  of 
the  parts  can  located  quickly,  e.g.,  a pump  must 
arrive  with  hoses  and  the  proper  couplings  in 
order  for  it  to  be  useful. 

Real-time  location  of  all 
equipment  registered 
with  CAD  design  of  site 

Medium 

By  knowing  the  precise  location  of  equipment 
with  respect  to  the  design,  time  can  be  saved  in 
many  tasks,  e.g.,  compact  the  area  around  lane 
joints  to  a greater  extent  than  the  remainder  of 
the  road  surface. 

Survey  integrated  with 
plan  from  highway 
department 

Short 

Moment  by  moment  guidance  can  be  sent  to 
operators  of  equipment  to  enhance  performance, 
e.g.,  graders  minimize  the  removal  of  earth  to 
achieve  the  goal  in  real-time. 

Short 

Minimize  rework  resulting  from  errors  in  read- 
ing the  blueprints  for  the  site  design.  Measure- 
ments can  be  made  (semi)automatically. 

Timely  distribution  of 
design  changes  accessi- 
bly from  design  database 

Short 

Minimize  rework  due  to  changes  in  design. 

8 


Table  1:  Productivity  improvement  through  hierarchical  control 


CAPABILITY 

TIME  FRAME 

EXAMPLE 

Improve  efficiency  of 
interaction  between  man 
and  equipment 

Short 

Moment  by  moment  guidance  and  instructions 
to  machine  operators  to  prevent  errors,  and 
maximize  quality  of  work.  Operators  can  have 
complete  information  — where,  when,  how 
much  — using  inexpensive  visual  displays. 

Short 

Operator  of  backhoe  can  use  CAD  model  of 
location  of  various  utilities  by  using  real-time 
display  of  digging  operations  to  avoid  cutting 
through  lines. 

Medium 

Control  system  on  backhoe  prevents  operator 
from  digging  into  volumes  occupied  by  utility 
lines. 

Reduce  time  need  for 
“cut  and  try”  situations 

Medium 

Automatic  calculation  of  the  required  dimen- 
sions for  materials  can  be  recomputed  in  real- 
time for  “as-is”  condition. 

Improve  efficiency  of 
interaction  between 
machine  and  machine 

Short 

Operator  follows  instructions  on  screen  to  per- 
form mating  of  machine  and  its  feeder. 

Medium 

Operator  drives  feeder  based  on  instructions  but 
all  other  mating  and  feeding  operations  are 
automatic. 

Long 

Machine  controls  all  aspects  of  itself  and  its 
feeder,  from  requesting  the  time  of  feeding,  to 
the  actual  coupling. 

Automatic  recovery  of 
as-built  databases 

i 

Medium 

Minimizes  cost  and  time  if  renovation  is  per- 
formed at  site  at  some  future  time. 

Increase  safety 

Medium 

Sensors  on  machines  monitor  the  state  of  the 
machine,  detect  errors,  and  shut  down  equip- 
ment in  an  orderly  fashion  when  a dangerous 
situation  occurs  or  a dangerous  action  is 
requested  by  the  operator. 

6.  Summary 

Hierarchical  control  lends  itself  to  the  organization  of  complex  systems  by  providing  the  glue 
to  connect  myriad  technologies  into  a coherent  and  understandable  system.  The  concepts  of  hier- 
archical control  were  presented  along  with  examples  of  specific  benefits  resulting  from  the  tech- 
nology. 
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The  project  described  here  will  represent  a demonstration  of  several  major  innovations  in  the 
integration  of  technologies  for  the  planning,  coordination,  and  control  of  activities  on  a highway 
construction  site.  The  resulting  improvements  should  fundamentally  affect  the  way  in  which  a con- 
struction project  can  be  managed,  and  will  result  in  more  efficient  use  of  manpower,  equipment 
and  materials,  as  well  as  increases  in  safety  and  quality  of  the  project.  The  demonstration  project 
will  integrate  a wide  variety  of  technologies  in  a manner  which  demonstrates  their  utility.  Many  of 
these  technologies  will  have  substantial  impact  in  themselves,  and  the  incremental  improvements 
which  result  from  adaptation  of  specific  individual  innovations  will  be  of  substantial  value.  For  ex- 
ample, improved  knowledge-based  planning  tools  will  be  valuable  even  with  conventional  work- 
site organization.  Similarly,  improved  standards  for  CAD  interfaces,  and  well-defined  networking 
protocols  for  worksites  will  have  substantial  benefits  in  the  gradual  improvement  of  worksite  tech- 
nology. 
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SENSOR  WORLD  TASK 

PROCESSING  MODELING  DECOMPOSITION 
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Figure  1.  Elements  of  the  RCS  Architecture 
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Figure  2.  Equipment  Is  Coordinated  by  the  Real-Time  Planner 
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Figure  3.  Sensing  the  Worksite 
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Figure  4.  RCS  for  Site  Integration  - Instructions  to  Individuals 
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ACTIONS 


PETRI  NET  MODEL 
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Figure  7.  Petri  Net  Model  Notation 


PETRI  NET  MODEL 
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Figure  8.  Petri  Net  Model  for  Trenching  and  Pipelaying 


1.  Objective 

SITE  INTEGRATION  PROPOSAL 

Outline  and  Deliverables 

The  overall  goal  is  ro  develop  automated  systems  which  will  reduce 
construction  time,  reduce  costs,  improve  efficiency  and  productivity,  and 
enhance  quality. 

2.  Approach 

Site-integration  management  systems  will  be  developed  which  will  integrate 
construction  databases  and  CAD  designs  with  interactive  planning  and 
scheduling,  and  provide  real-time  control  of  on-site  activities. 

3.  Deliverables 


Phase  1: 

An  in-depth  study  of  an  example  bridge-construction  site,  and 
plan  for  development  of  the  site-management  i«»ols.  Time:  1 
Year. 

Phase  2: 

A site-management  simulator  with  full  plannin-  and 
scheduling  capability,  resource  allocation,  contingency 
handling,  and  site  visualization.  Construction  managers  and 
supervisors  will  be  presented  with  all  options  and  constraints 
and  will  be  able  to  see  consequences  of  alternative  decisions  on 
resource  allocation,  schedules,  and  subsequent  alternatives. 

The  state  of  the  site  at  any  point  in  time,  under  any  set  of 
decisions  will  be  visible  3D  displays.Time:  1.5  Years. 

Phase  3: 

Sensing  and  control  interfaces  will  be  added  to  the  simulator 
enabling  it  to  accept  real-time  site  data  and  to  interface  to 
supervisors  and  equipment  operators.  This  system  will  then  be 
tested  in  construction  of  a bridge  site  comparable  to  the  site 
studied  in  Phase  1.  Time:  1 Year. 
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SITE  INTEGRATION  PROPOSAL 
Timeline  of  Activities 
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Project  Management 


SITE  INTEGRATION  PROPOSAL 
Planning  and  Scheduling  Activities 

PLANNING  - What  needs  to  be  done  (activities)  and  how  long  will  it  take 
(activity  time)  with  a given  level  of  resource  allocation  (time,  material, 
equipment,  workforce). 

SCHEDULING  - When  and  in  what  sequence  activities  will  be  performed  to 
satisfy  dependence  and  constraints. 

CONTROL  - Monitor  the  execution  of  the  activities  and  adjust  the  plan,  schedule, 
and  resource  allocation  to  improve  outcome. 

SCHEDULE  OF  WORK 

YEAR  1 : Integrate  database  and  graphical  display  of  site  operations  with  C P M 
activity  network  to  visualize  plans. 

- Incorporate  existing  commercial  CPM  software 

- Test  and  evaluate  on  phase  l bridge  study 

YEAR  2:Integrate  resource  allocation  tools  into  activity  planning  —interactively 
recommend  alternative  plans  and  visualize  resulting  operations  and  schedules. 

- Utilize  existing  compiled  data  and  guidelines  where  possible 

- Analyze  and  incorporate  results  of  the  phase  1 bridge  study 

- Focus  on  recommendation  of  likely  choices  with  interactive  decision  making 
rather  than  optimization  and  automated  selection. 

YEAR  3:  Implement  framework  for  integration  of  site  data  and  monitoring  to 
update  plan  and  support  decision  making. 

- Extend  links  of  integrated  database  to  update  activities  and  states 

- Develop  mapping  from  activity  network  to  Petri  net  model  (use  existing  Petri 
net  simulation  and  analysis  software) 

- Incorporate  branching  transitions  and  resource  related  preconditions 

- Incorporate  Petri  net  analysis  tools  for  safeness,  liveness  boundedness 

YEAR  4:  Support  bridge  construction  case  study  using  integrated  planning  tools. 

- Track  plans  and  schedules  versus  execution  of  operations 

- Conduct  analysis  of  alternatives  in  parallel  with  execution 

- Study  and  modify  planner  interfaces  as  the  project  proceeds. 

- Summary  and  report  on  project  — evaluation  of  planner  capabilities  and  impact 
on  project  outcome. 
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SITE  INTEGRATION  PROPOSAL 
Simulator  Prototype  Activities 


The  purpose  of  the  site  integration  simulator  prototype  is  to  store  a 
representation  of  the  worksite  (the  world  model),  to  keep  the  representation 
current  with  processed  sensor  data,  to  interface  the  representation  to  application 
programs  (GIS/  CAD/  planning/  scheduling),  and  to  provide  an  interactive 
graphical  user  interface. 

Tasks: 

1.  Establish  functional  requirements: 

- survey  bridge-building  handbooks  and  guides 

- review  data  gathered  at  bridge  worksite 

- identify  construction  site  objects  to  be  modelled 

- identify  characteristics  of  the  objects  to  be  modelled 

- identify  interactions  between  objects  to  be  modelled 

- identify  RCS  requirements 

2.  Select  appropriate  information  technology: 

- survey  existing  GIS/CAD  systems  and  database  development  environments 

- select  system(s)  to  be  used  in  prototype  based  on  functional  requirements 

3.  Design  site  integration  simulator  prototype: 

- define  conceptual  objects/schemas 

- define  program  interfaces  to  GIS/CAD  systems  and  to  planning  and  scheduling 
components 

- define  functionality  of  the  interactive  graphical  user  interface 

4.  Implement  basic  site  integration  simulator  prototype: 

- build  database 

- build  interfaces  to  GIS/CAD  application  programs 

- populate  and  test  with  data  gathered  at  worksite 

5.  Implement  planning  and  scheduling  in  the  site  integration  simulator  prototype: 

- build  interfaces  to  planning  and  scheduling  components 

- test  by  simulating  observed  worksite  activities 
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6.  Implement  interactive  graphical  user  interface: 

- build  program  elements  that  provide  designed  functionality 

- test  and  modify  interface  based  on  in-lab  experiments  with 
engineering/construction  personnel 

7.  Harden  prototype  and  implement  second  bridge  project. 

The  proposed  manpower  and  funding  for  these  tasks  is  3 FTEs  for  the  first  three 
years  at  $450K/year,  and  1 FTE  for  the  fourth  year  at  $150K. 

At  a minimum  level  of  effort  (150K),  only  task  1 and  the  first  element  of  task  3 
would  be  performed  and  a report  would  be  issued. 
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Site  Integration  Simulator  Prototype 
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Integrated  world  model 


SITE  INTEGRATION  PROPOSAL 
Real-Time  Control:  Activities 


Phase  1 : 

- Develop  interfaces  for  RCS  to  support  planning  representation  and 
command. 

- Observe  the  bridge  site.  Study  the  ways  an  operator  expects  to  use 
the  system.  Study  sensing  problems. 

- Plan  implementation  of  RCS  instantiation  and  coordinate  with 
planning  and  database  system  to  ensure  compatibility. 

Phase  2: 

- Implement  RCS  installation  and  integrate  it  with  simulation 
capabilities. 

- Implement  prototype  operator  interface  system. 

- Develop  operator  interface  screens  for  different  modes  of 
operation. 

- Experiment  with  integration  of  commercially  available 
measurement  and  sensor  systems  . 

Phase  3: 

- Harden  implementation  of  RCS  and  test  thoroughly. 

- Support  operations  during  bridge  construction. 

- Analyze  performance. 


SITE  INTEGRATION  PROPOSAL 
Schedule  of  On-Site  Events 


Bridge-site  Observation  Phase. 

One  technical  worker  would  be  assigned  to  "live  with"  the  bridge.  He  would 
spend  essentially  full  time  on  site.  His  duties  would  be  to  become  fully  familiar 
with  all  phases  of  the  construction  process  so  that  he  could  subsequently  serve  as  a 
resource  to  the  scientific  team  developing  the  site  integration  tools.  He  would 
require  the  cooperation  of  the  supervisors  and  workers  on  the  site  to  answer 
questions  and  explain  procedures,  but  would  be  instructed  to  minimize 
interference  with  ongoing  activities.  He  would  need  to  be  able  to  videotape 
operations  for  later  review  by  project  scientists.  He  would  need  on  occasion  to  set 
up  tests  of  sensing  equipment  to  study  the  performance  of  different  sensors  under 
site  conditions.  Also,  runs  of  data  from  sensors  would  be  required  to  serve  as 
simulated  input  during  later  development  of  the  computer  programs.  Other 
technical  staff  from  the  cooperating  scientific  groups  would  need  to  visit  the  site 
to  see  specific  things.  The  on-site  technical  worker  would  be  responsible  for 
arranging  their  visits  to  coincide  with  the  work  to  be  observed,  and  for  setting  up 
any  special  tests  needed  for  their  visits.  In  general  no  significant  interference  with 
construction  personnel  or  equipment  is  foreseen.  A small  van  would  be  provided 
to  house  all  equipment,  and  any  sensors  placed  on  equipment  or  in  work  areas 
would  be  small  and  would  be  fully  cleared  with  operators  and  supervisors  before 
installation. 

Bridge  Construction  Phase. 

After  development  of  the  site-integration  tools  and  installation  of  the  sensors  at 
the  test  site,  we  would  expect  the  following  scenario. 

All  materials  entering  the  site  and  moving  within  the  site  would  be  tracked  by  the 
system  through  the  use  of  GPS/GIS  technology  and  other  sensors.  The  system 
would  employ  site-planning  and  resource  allocation  techniques  to  specify  the 
optimal  placement  of  materials  and  timing  and  routing  of  materials.  The  current 
state  of  die  site  and  the  expected  future  state  at  any  time  could  be  viewed  by 
supervisory  or  management  personnel  in  3D  displays  with  color-coded  indication 
of  status  of  completion.  As-built  databases  would  be  automatically  generated,  and 
any  differences  between  planned  and  as-built  status  would  be  highlighted. 
Management  decisions  would  be  translated  into  schedules  for  supervisors'  which 
would  be  coordinated  to  achieve  optimal  efficiency  in  use  of  crews  and 
equipment.  Contingencies  such  as  weather  or  machinery  breakdown  that  forced 
changes  in  scheduling  would  be  accepted  by  the  system.  It  would  then  give 
managers  and  supervisors  a display  of  the  scheduling  consequences,  and  plans  for 
optimal  ways  to  change  the  work  schedule. 


SITE  INTEGRATION  PROPOSAL 
Proposed  Costs  By  Year 


Time  Schedule: 


On-site  study  of  first  bridge  construction  1.0  yr. 

Construction  of  database  and  simulator  1 .5  yr. 

Interface  to  sensor  and  control  systems  0.5  vr. 

3.0  yr. 

Hardening  and  testing  while  waiting  lor 

2nd  site  to  commence  construction.  0.0  to  0.5  vr. 


0.0  to  0.5  yr. 

Test  integration  of  2nd  bridge  construction  1 .0  yr. 

Study  and  analysis  of  results  0.5  vr 

1.5  yr. 

Total  time  4.5  to  5.0  yrs. 


Budget,  year  one: 
Manpower,  fully  loaded 


RPI  0.2  faculty,  0.5  graduate  student  75k 

BFRL  0.5  FTE  staff  75k 

RSD  0.5  FTE  staff  75k 

On-site  staff  plus  expenses  and  support  225k 

450k 

Technology  assessment  study  25k 

Project  managen*  ot  and  software  25k 

50k 

Total,  year  one:  500k 


Budget,  year  two: 

Manpower,  fully  loaded 


RPI  0.5  faculty,  3.0  graduate  students  230k 

BFRL  3.0  FTE  staff  450k 

RSD  2.0  FTE  staff  300k 

980k 

Project  management  100k 

Software  150k 

250k 

Total,  Year  Two:  1230k 

Budget,  year  three: 

Manpower,  fully  loaded  300k 

BFRL  2.0  FTE  staff  500k 

RSD  5.0  FTE  staff  (2.0  full-time  on  site)  225k 

1025k 

Project  management  100k 

Terminals,  networks,  interfaces,  sensors  250k 

Field  testing  and  hardening  75k 

425k 

Total,  year  three:  1450k 

Six  month  period 

(Allowed  for  synchronization 

with  2nd  bridge  project  (Est.)  and  additional 

preparation  as  indicated.) 

Site  setup  25k 

Field  testing/hardening  75k 

Total  100k 

Budget,  year  four: 

Manpower,  fully  loaded 

RPI  0.2  Faculty,  1.0  Graduate  Student  150k 

BFRL  1.0  FTE  Staff  150k 

RSD  1.0  On-site  staff  + support  230k 


100k 


Project  Management  and  travel 


Total,  year  four 

630k 

Budget,  Six-months  of  final  analysis 

Analysis  of  results 

100k 

Management,  report  preparation 

25k 

Total 

125k 
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SUMMARY 


Distress  assessment  is  an  essential  component  of  pavement 
management  systems.  The  objectives  of  distress  assessment  are  to 
evaluate  the  performance  of  pavement  structures,  monitor  pavement 
condition  over  time  and  to  formulate  appropriate  maintenance 
programs . 

There  are  no  standard  procedures  for  conducting  distress 
surveys  at  the  current  time.  These  methods  involve  a significant 
manual  component  that  results  in  a number  of  problems,  such  as 
non-repeatability,  subjectivity  and  high  personnel  costs. 
Consequently,  it  has  been  recognized  that  automating  distress 
surveys  using  image  analysis  is  of  great  significance  in 
objectively  evaluating  the  distresses. 

The  present  study  addresses  the  development  of  a computerized 
system  for  automating  distress  data  based  on  mathematical 
morphology  that  uses  the  texture  present  on  the  pavement  surfaces 
to  analyze  cracking  as  well  as  other  forms  of  distresses.  The  goal 
of  the  proposed  study  is  to  develop  and  refine  morphology-based 
image  analysis  algorithms,  and  evaluate  the  feasibility  of  applying 
this  computerized  technology  to  the  automation  of  pavement  distress 
surveys.  Findings  will  be  based  on  the  ability  of  the  developed 
image  processing  technique  to  identify  distress  types  and 
severities  on  the  photographs  of  asphalt  and  concrete  pavement 
surfaces. 

A significant  feature  of  the  morphological  approach  is  that 
distresses  other  than  cracking  may  also  be  detected  and  quantified. 
Another  major  feature  of  the  proposed  study  is  that  it  does  not 
involve  the  development  of  different  algorithms  to  detect  and 
quantify  different  distresses.  The  development  of  a unified 
approach  to  handle  different  types  of  distresses  under  different 
conditions  leads  to  a significant  computational  advantage. 
Finally,  the  current  approach  is  capable  of  performing  well  on  a 
number  of  different  pavement  surfaces,  including  those  that  exhibit 
extensive  texture.  Unlike  other  existing  approaches,  the  current 
algorithm  utilizes  texture  information  and,  therefore,  is  more 
robust  in  the  presence  of  pavement  texture. 


2. 


INTRODUCTION 


Distress  assessment  is  an  essential  component  of  pavement 
management  systems.  The  objectives  of  distress  assessment  are  to 
evaluate  the  performance  of  pavement  structures,  monitor  pavement 
condition  over  time  and  to  formulate  appropriate  maintenance 
programs.  The  goal  of  the  proposed  research  effort  is  to  study  the 
feasibility  of  automating  pavement  distress  surveys  and  the 
development  of  a computerized  image  processing  system  for 
conducting  distress  surveys.  This  study  represents  the  work  being 
done  at  Rensselaer  Polytechnic  Institute  in  collaboration  with  the 
New  York  State  Thruway  Authority. 

There  are  no  standard  procedures  for  conducting  distress 
surveys  at  the  present  time.  Available  methods  vary  in  cost  as  well 
as  data  quality  and  range  from  manual  field  mapping  to  visual 
surveys  with  automated  data  logging.  These  methods  involve  a 
significant  manual  component  that  results  in  a number  of  problems, 
such  as  non-repeatability,  subjectivity  and  high  personnel 
costs [6],  Consequently,  it  has  been  recognized  that  automating 
distress  surveys  using  image  analysis  is  of  great  significance  in 
objectively  evaluating  the  distresses. 

A direct  approach  to  automating  distress  procedures  is  via 
sensed  images,  either  digitized  photographs  or  video  images,  of 
pavement  surfaces.  An  advantage  of  using  digitizing  cameras  to 
generate  a reflected  image  of  the  pavement  surface  is  that  the 
observed  features  can  be  related  to  those  recognized  by  the  human 
eye.  Mathematical  morphology  is  an  image  processing  technique  that 
was  born  out  of  investigation  of  the  relationships  between  the 
geometry  of  the  porous  media  and  their  permeabilities.  Thus,  it  has 
its  roots  in  the  determination  of  material  properties  and  defines 
powerful  tools  for  texture  analysis [9].  Pavement  structures  are 
considered  as  composite  materials  with  particles  (aggregates)  bound 
together  by  asphalt  or  Portland  cement  concrete  mix.  Analysis  of 
pavement  textures  (which  result  from  such  a composite  structure) 
could  reveal  information  useful  in  the  assessment  of  their  physical 
condition,  and  such  analysis  provides  the  rationale  for  exploring 
the  applicability  of  the  methods  available  in  mathematical 
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morphology. 


The  present  study  addresses  the  development  of  a computerized 
system  for  automating  distress  data  based  on  mathematical 
morphology  that  uses  the  texture  present  on  the  pavement  surfaces 
to  analyze  cracking  as  well  as  other  forms  of  distresses.  The  goal 
of  the  proposed  study  is  to  develop  and  refine  morphology-based 
image  analysis  algorithms,  and  evaluate  the  feasibility  of  applying 
this  computerized  technology  to  the  automation  of  pavement  distress 
surveys.  Findings  will  be  based  on  the  ability  of  the  developed 
image  processing  technique  to  identify  distress  types  and 
severities  on  the  photographs  of  asphalt  and  concrete  pavement 
surfaces. 

The  remaining  text  is  organized  as  follows.  Section  3 
discusses  several  existing  image  processing  systems  for  automating 
distress  condition  assessment,  and  also  establishes  the  context  of 
the  proposed  study.  Section  4 presents  the  proposed  research  method 
and  discusses  the  research  efforts  to  date.  Section  5 summarizes 
the  relevant  task  plan. 

3 . BACKGROUND 

The  availability  of  high-speed  computers,  video  and 
photographic  cameras,  and  image  processing  software  has  resulted  in 
the  development  of  a number  of  early  systems  for  automating 
pavement  distress  surveys  using  computerized  image  processing. 

Automated  Road  Image  Analyzer  (ARIA)  is  a computerized  image 
analysis  system  developed  by  MHM  Associates  Inc. [8],  PAVEDIST 
developed  by  Acosta  et.  al.[l]  and  PAS  1 developed  by  Pavedex 
Inc.  [5]  are  examples  of  computerized  image  processing  systems  for 
automated  crack  detection.  They  demonstrated  the  ability  to 
quantify  alligator,  longitudinal  and  transverse  cracks.  The  basic 
idea  involves  identifying  cracks  by  their  different  reflectivity 
properties.  Thus,  image  regions  are  separated  into  different 
classes  where  one  of  the  classes  corresponds  to  distresses.  Area, 
transverse  and  longitudinal  lengths,  aspect  ratio,  gray  level  mean 
and  variance,  inertia,  etc.,  are  some  of  the  properties  used  in  the 
classification.  Their  performance  was  compared  against  that  of 
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human  raters  and  accuracies  up  to  90%  were  reported  on  certain 
types  of  pavements,  especially  when  there  is  no  significant  amount 
of  texture. 

The  several  image  processing  systems  that  already  exist 
clearly  demonstrate  the  potential  for  automated  processing. 
However,  the  majority  of  them  use  relatively  simple  algorithms  that 
can  only  support  crack  detection,  and  thus  are  not  capable  of 
evaluating  other  types  of  distresses  such  as  spalling  and  surface 
defects.  All  are  subject  to  difficulties  in  the  presence  of 
texture.  In  an  assessment  of  the  leading  technologies  in  [7],  it 
is  reported  that  it  is  quite  difficult  to  segregate  cracks  from 
texture,  particularly  for  the  more  open  textures  of  the  bituminous 
pavements.  It  was  also  concluded  that  the  existing  techniques 
based  on  thresholding  will  generally  identify  some  texture  as  well 
as  cracks,  especially  in  scenes  which  have  no  cracks.  Thus, 
computer  identification  of  pavement  distresses  from  digitized 
photographs  or  video  images  requires  the  system  to  perform  equally 
well  under  a variety  of  different  conditions  and  requires  fairly 
sophisticated  image  processing  algorithms. 

4 . RESEARCH  METHOD 

4.1  Overview 

The  present  study  addresses  the  development  of  a different 
approach  to  automating  distress  data,  one  based  on  mathematical 
morphology  that  uses  the  texture  present  on  the  pavement  surfaces. 

A significant  feature  of  the  morphological  approach  is  that 
distresses  other  than  cracking  may  also  be  detected  and  quantified. 
The  proposed  normalizing  scheme  to  incorporate  a-priori  knowledge 
about  non-distressed  pavements  increases  the  sensitivity  of 
particle  distributions  to  all  types  of  distresses.  This  increased 
sensitivity  contrasts  with  the  existing  image  processing  techniques 
that  do  not  handle  such  distresses.  Another  major  feature  of  the 
proposed  study  is  that  it  does  not  involve  the  development  of 
different  algorithms  to  detect  and  quantify  different  distresses. 
The  development  of  a unified  approach  to  handle  different  types  of 
distresses  under  different  conditions  leads  to  a significant 
computational  advantage.  Finally,  the  current  approach  is  capable 
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of  performing  well  on  a number  of  different  pavement  surfaces, 
including  those  that  exhibit  extensive  texture[2,4] . Unlike  other 
existing  approaches,  the  current  algorithm  utilizes  texture 
information  and,  therefore,  is  more  robust  in  the  presence  of 
pavement  texture. 

4.2  Mathematical  Morphology 

Mathematical  morphology  (MM)  is  an  image  processing  technique 
that  was  born  out  of  investigation  of  the  relationships  between  the 
geometry  of  porous  media  and  its  permeability.  MM  can  be  used  to 
assess  geometric  features  such  as  size,  orientation,  shape, 
overlap,  and  distributions  of  objects  in  photographic  or  video 
images.  The  power  of  this  technique  is  based  on  a structural 
sorting  of  texture.  Texture  is  analyzed  through  the  use  of 
geometric  probes  known  as  "structuring  elements."  When  different 
shapes  and  sizes  of  these  elements  are  used,  different  features  of 
interest  are  highlighted.  For  example,  a circular  structuring 
element  of  a specific  diameter  'd'  sorts  the  texture  under  study  so 
that  the  structures  possessing  a similar  geometry  (i.e.,  circles  of 
diameter  'd')  are  emphasized. 

The  MM  approach  is  somewhat  analogous  to  sieving  a soil 
sample.  Probing  (or  "opening")  an  image  with  an  element  has  the 
effect  of  removing  from  the  image  all  objects  smaller  than  the 
structuring  element.  Performing  a series  of  openings  with 
increasingly  larger  structuring  elements  of  the  same  shape  is 
similar  to  shaking  a soil  sample  through  a series  of  sieves  with 
decreasing  mesh  openings.  The  generated  morphological  "opening 
distribution"  is  analogous  to  the  amounts  of  soil  retained  on 
screens  of  various  sizes.  Thus,  the  opening  distribution  provides 
important  information  about  the  size  distribution  of  objects  in  an 
image . 


4.3  Crack  Assessment 

The  MM  approach  for  crack  assessment  is  illustrated  in  Figure 
1 which  shows  a thresholded  pavement  image  along  with  its  opening 
distribution.  The  opening  distribution  is  obtained  by  using  a 
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horizontally  oriented  line-shaped  structuring  element.  The  length 
of  the  line  segment  is  varied  from  a length  of  2 pixels  to  4 6 
pixels.  From  the  description  of  the  opening  operation,  one  can 
qualitatively  describe  the  shape  of  the  distribution  in  the  figure. 
There  is  a fine-grain  texture  in  the  original  image  caused  by 
particles  having  dimensions  of  about  8 pixels  or  less.  There  are 
also  two  major  cracks  running  vertically  in  the  image  that  have 
much  greater  widths  (approximately  26  to  32  pixels) . Consider 
opening  the  image  with  a series  of  line-shaped  structuring  elements 
of  successively  increasing  lengths.  The  fine-grain  texture  is 
continuously  removed  by  opening  with  lines  of  length  less  than  8 
pixels.  Thus,  the  opening  distribution  exhibits  a large  slope  at 
those  scales.  Since  a line-shaped  structuring  element  fits  inside 
cracks  (which  are  also  long  features) , it  preserves  cracking  until 
its  length  becomes  longer  than  that  of  the  crack.  Structuring 
elements  of  lengths  in  the  range  26  to  32  pixels  begin  to  remove 
the  cracking  from  the  image  which  results  in  the  sharper  slopes  at 
those  scales  in  the  distribution. 

4.4  Surface  Distress  Assessment 

The  qualitative  analysis  described  above  shows  that  an  opening 
distribution  generated  using  a linear  structuring  element  provides 
information  about  cracking.  Relatively  large  defects  such  as 
potholes  can  be  analyzed  similarly.  By  "normalizing"  opening 
distributions  to  remove  the  effect  of  normal  pavement  surface 
texture,  distresses  such  as  polishing,  raveling,  pitting,  etc.,  can 
also  be  detected.  Without  normalization,  texture-related 
distresses  can  be  difficult  to  identify,  because  there  is  often  no 
clear  difference  between  the  scale  of  normal  pavement  texture  and 
that  of  the  abnormal  condition.  Normalizing  emphasizes  all  types 
of  deviations  from  normal  surface  texture,  and  therefore  can  also 
be  expected  to  increase  sensitivity  of  crack  detection  (see  [2]  for 
details)  . 

The  shape  of  a normalized  distribution  computed  from  a 
pavement  image  with  no  distresses  is  always  a straight  line 
parallel  to  the  X-axis  at  y = 1.  Any  abnormal  texture  is  revealed 
as  a deviation  from  the  straight  line;  if  there  is  an  excess  of 
particles  at  any  scale,  then  it  is  greater  than  1,  and  if  there  is 
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A(X) 


Cel  i 

Figure  1:  (a)  Original  image.  Results  of  opening  it  with  a 
line  shaped  structuring  element  of  length  (b)  8,  (c)  30 
pixels,  (d)  Opening  Distribution.  X-axis:  length  of  the 
structuring  element;  Y-axis:  area  remaining  in  the  image  after 
opening  in  pixels. 

a deficiency  at  any  scale,  it  is  less  than  1.  In  an  opening 
distribution  generated  from  line-shaped  structuring  elements, 
cracks,  which  are  large  scale  features,  are  revealed  as  overshoots 
in  the  normalized  distribution  at  corresponding  scales.  Abnormal 
texture  resulting  from  other  types  of  distresses  is  revealed  as 
either  an  undershoot  or  an  overshoot  at  scales  corresponding  to  the 
scales  of  distress.  Full  mathematical  details  are  presented  in  [2]. 


Figure  2 shows  the  normalized  distributions  of  four  images 
showing  shoulder  defects.  The  images  are  identified  by  an 
alphabetic  code  of  the  form  "ts_d.dd"  for  brevity.  The  first 
character  ("t" ) indicates  the  type  of  distress;  in  this  example, 
all  photos  are  coded  "g,"  and  portray  shoulder  defects.  The  second 
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Figure  2:  (a)  Four  pavement  shoulder  images  with  defects,  (b) 
Normalized  Distributions  of  the  four  images.  Structuring 
element  is  a line.  X-axis  is  the  length  of  the  structuring 
element.  Y-axis  is  the  normalized  area  (dimensionless) 


character  "s"  indicates  the  distress  severity  as  assessed  by 
trained  manual  raters.  Severity  may  be  none  (N)  , small  (S) , medium 
(M)  , large  (L)  , or  total  (T)  , as  defined  for  the  New  York  State 
Thruway  Authority's  pavement  distress  survey  [3].  The  digits 
following  the  underscore  are  used  to  identify  an  image  uniquely. 
Thus,  the  image  "gn_2.31"  indicates  it  is  a particular  image  of  a 
pavement  shoulder  exhibiting  no  defects.  In  Figure  2,  the 
normalized  distribution  of  "gn"  image  is  almost  a straight  line 
parallel  to  the  Y-axis  at  y = 1.0  showing  that  it  has  an  ideal 
pavement  surface.  The  rest  of  the  distributions  show  deviations 
from  the  flat  line  of  varying  magnitudes  at  varying  scales.  The 
"gs"  and  "gm"  images  exhibit  cracking  and  their  normalized 
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distributions  reveal  it  as  overshoots  at  large  scales  (26  and  32 
pixels) , as  expected  from  the  discussion  in  the  previous  paragraph. 
The  magnitude  of  the  deviations  for  the  "gm"  image  is  larger  than 
that  of  the  "gs"  image  indicating  the  greater  severity  of  its 
cracking,  "gl”  image  shows  a large  overshoot  at  a scales  between  4 
and  12  pixels.  It  does  not  show  any  deviations  at  larger  scales 
indicating  that  it  has  little  or  no  cracking,  and  the  image 
ngl_2 . 37"  shows  that  the  distress  is  present  mostly  as  a surface 
defect.  Thus,  normalized  distributions  detect  and  quantify  cracking 
as  well  as  other  types  of  distresses [2] . 

5 . TASK  PLAN 

Different  morphological  models  of  ideal  pavement  textures  are 
being  developed  to  study  their  use  in  identifying  pavement 
distresses.  The  preliminary  results  show  that  the  morphological 
approach  based  on  particle  size  distributions  is  capable  of 
detecting  and  quantifying  cracking?  it  also  provides  information 
about  the  overall  condition  of  the  pavement,  including  non-cracking 
distresses.  The  results  indicate  the  potential  of  morphological 
size  distributions  for  analyzing  pavement  surface  images. 

Automating  distress  condition  assessment  using  computerized 
image  processing  is  a two-step  process.  In  the  first  stage,  images 
of  the  pavement  surface  are  analyzed  to  derive  certain  measurements 
in  the  image  domain.  The  second  stage  involves  translating  these 
measures  into  the  pavement  domain.  Most  of  the  research  effort  to 
date  has  been  related  to  the  first  stage  and  the  normalized 
distributions  represent  the  resulting  image  measures.  The  proposed 
tasks  represent  the  translation  procedure  of  the  second  stage. 

Specifically,  over  the  one-year  period  of  the  study,  the 
feasibility  investigation  would  involve  the  following  tasks: 

1)  Measure  the  overall  severity  of  distresses 

The  overall  severity  is  used  to  eliminate  images  that  have  no 
significant  distress  from  the  analysis.  Precise  identification 
of  different  types  of  distresses  is  not  required.  This 
information  can  be  obtained  by  relating  the  magnitude  of  the 
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deviations  in  the  normalized  morphological  distributions  with 
the  severity  of  the  distresses. 

2)  Classification  and  quantification  of  cracking 

Analysis  of  the  image  measures  obtained  from  the  morphological 
distributions  to  relate  them  to  longitudinal,  transverse  and 
alligator  cracking.  These  different  classes  are  based  on  the 
direction  of  cracking.  Such  information  may  be  obtained  by 
using  line-shaped  structuring  elements  oriented  in  different 
directions  to  generate  normalized  distributions.  The 
normalized  distributions  may  also  be  analyzed  to  quantify 
different  types  of  cracking  according  to  their  severity  and 
extent . 

3)  Assessment  of  other  Surface  Defects 

Analyze  the  morphological  distributions  and  relate  them  to 
non-cracking  distresses  like  spalling,  surface  defects, 
milling  etc. 

4)  Generate  a numerical  index  for  the  overall  condition 

Combine  the  information  obtained  from  the  earlier  tasks  to 
generate  a numeric  index  that  can  be  related  to  the  overall 
condition  of  the  pavement.  Different  weights  may  be  given  to 
different  types  of  defects  depending  on  their  importance. 


Tasks  / Months 

123456789  10  11  12 

Task  1 

Task  2 

Task  3 

Task  4 

Table  1:  Task  Schedule 
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BUDGET 


Rough  budget  for  1 year  of  support. 

Automated  pavement  distress  assessment  using  morphological  image  analysis. 

Co-PI:  Dimitri  A.  Grivas,  Associate  Professor,  Civil  Engineering,  RPI 
Co-PI:  Michael  M.  Skolnick,  Associate  Professor,  Computer  Science,  RPI 

We  estimate  that  the  following  level  of  support  will  come  to  a bottom  line  figure 
(including  fringe  and  overhead)  of  about  $100,000. 

D.  Grivas:  10%  academic  year  salary  and  1 month  summer 

M.  Skolnick:  20%  academic  year  salary  and  1 month  summer 

One  R.A.:  Coverage  for  tuition,  salary  over  academic  year  and  summer. 

Other  expenses  such  as  computer  time,  secretarial  coverage,  supplies  and 
domestic  travel. 
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A Proposal  on  Automated  Crack  Detection  and  Sealing 


Avi  Kak 

Robert  Cromwell 

Robot  Vision  Lab 
Purdue  University 


Introduction 

We  believe  that  the  time  has  come  for  a concerted  effort  at  applying  the  science  and  tech- 
nology of  automation  to  the  automation  of  road  surface  inspection  and  repair.  This  proposal  has 
two  objectives:  1)  To  develop  the  technology  of  automated  inspection  of  road  surfaces;  and  2) 
To  develop  a prototype  of  a mobile  robotic  system  that  can  be  hitched  to  a maintenance  vehicle 
traveling  at  approximately  10  miles/hour  for  automated  sealing  of  the  cracks  found.  The  data 
produced  through  inspection  would  be  analyzed  automatically  for  flagging  the  sections  of  high- 
way in  need  of  repair.  In  addition,  each  damaged  section  of  the  highway  would  be  graded 
according  to  the  severity  of  the  cracks  and  other  manifestations  of  deterioration.  Furthermore, 
the  cracks  whose  severity  exceeds  a certain  treshold  will  be  sealed  automatically  by  the  robotic 
system.  The  design  of  the  automatic  sealing  arm  will  make  use  of  the  neural-network  and 
fuzzy-logic  based  kinematic  control  strategies  we  have  developed  recently  [PacMin93a, 
PacMin93b]. 

Why  Purdue  Robot  Vision  Lab 

During  the  last  decade,  the  Robot  Vision  Lab  (RVL)  at  Purdue  has  produced  some  of  the 
most  computationally  efficient  and  robust  algorithms  for  sensor-based  robotics.  (These  accom- 
plishments have  formed  the  basis  of  an  ongoing  active  collaboration  with  industry  in  the  area  of 
advanced  automation.)  To  state  two  of  our  most  significant  accomplishments  during  the  last  few 
years:  RVL  has  developed  the  fastest  3D  vision  system  for  bin-picking  [CheKak89,  KimKak91]. 
On  run-of-the-mill  laboratory  computing  hardware,  this  system  can  recognize  objects,  including 
those  that  are  non-polyhedral  and  non-convex,  and  compute  their  poses  in  just  a few  seconds. 
And,  RVL  has  developed  the  fastest  vision-based  mobile  robot  navigation  system  that  allows  a 
robot  to  navigate  indoors  at  speeds  of  9 to  12  meters  per  minute  using  computing  hardware  of  no 
more  than  16  MIPS  power  [KosKak92].  These  developments  have  been  made  through  the 
discovery  of  new  data  structures,  for  representing  objects  and  space,  and  of  new  control  stra- 
tegies for  robust  decision-making.  More  recently  the  laboratory  has  also  developed  neural- 
network  and  fuzzy-logic  based  nonlinear  control  strategies  for  kinematic  and  dynamic  control 
[PacKak93a,  PacKak93b]. 

Is  Robotic  Crack  Sealing  Feasible? 

Robotic  sealing  of  longitudinal  cracks  while  the  maintenance  vehicle  is  in  continuous 
motion  at  10  mph  has  already  been  demonstrated  by  the  California  Department  of 
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Transportation.  This  was  done  by  attaching  a conventional  robotic  arm  to  the  rear  of  the  vehicle 
and  using  a vision  system  to  guide  the  placement  of  a sealer  nozzle  on  the  crack.  While  evi- 
dently a wonderful  demonstration  in  the  use  of  robotics  and  vision  to  highway  maintenance,  the 
use  of  a conventional  robotic  arm,  designed  originally  for  pick-and-place  operations  in  a factory, 
meant  that  this  effort  was  preordained  to  not  succeed  for  the  sealing  of  transverse  cracks  with  the 
vehicle  in  motion. 

Evidently,  an  entirely  new  kind  of  a robotic  arm  is  needed  for  the  task.  In  Fig.  A,  we  have 
shown  a concept  drawing  of  such  an  arm,  although  variations  on  the  depicted  theme  are  certainly 
possible.  Basically,  the  arm  will  have  flixibility  to  scan  along  a transverse  crack  while  the  vehi- 
cle is  in  motion  longitudinally.  Assume  that  initially  we  want  to  design  a system  that  would 
allow  the  vehicle  to  move  down  the  road  at  10  mph.  This  speed  translates  into  176  inches  per 
second.  Now  let’s  assume  a lane  width  of  12  feet.  The  California  work  has  already  demonstrated 
that  the  viscosity  of  the  sealing  compound  is  such  that  the  nozzle  can  move  at  approximately  10 
mph,  meaning  at  176  inches  per  second,  while  ejecting  the  sealing  compound.  Therefore,  speak- 
ing conservatively,  it  will  take  up  to  one  second  to  fill  up  a transverse  crack. 1 During  this  inter- 
val, the  vehicle  will  experience  a forward  translation  of  176  inches.  The  question  then  becomes 
whether  it  would  be  possible  for  a robotic  arm  to  extend  itself  by  about  15  feet.  We  believe  that 
a scissor-shaped  kinematic  chain  depicted  in  the  figure  would  be  able  to  do  so.  Such  a chain 
would  be  mounted  on  a 20  foot  boom  that  would  be  anchored  at  its  free  end  on  top  of  wheeled 
support.  Note  that  for  actuation  along  the  boom,  only  the  last  link  of  the  scissor  chain  will 
be  coupled  with  the  threads  on  the  boom;  the  other  links  will  simply  ride  over  the  threads. 
Also  note  that  the  entire  boom  will  be  pivoted  by  a motor  installed  over  the  free  end;  the 
rotation  will  be  caused  by  the  turning  the  wheel  under  the  free  end  of  the  boom. 

An  obvious  shortcoming  of  the  mechanism  shown  in  the  figure  is  that  once  a transervse 
crack  is  filled  up,  the  arm  will  not  be  able  to  fill  up  any  further  transverse  cracks  until  the  it  is 
retracted.  Assuming  it  takes  another  second  to  retract  the  arm,  that  would  imply  that  the  vehicle 
would  not  be  able  to  fill  up  the  transverse  cracks  that  are  spaced  more  closely  than  30  feet. 
While  the  technology  is  still  in  the  development  phase,  a possible  way  to  circumvent  this 
difficulty  would  be  to  make  multiple  passes  on  a highway.  So,  if  the  cracks  are  relatively  infre- 
quent, with  fault-free  spaces  of  length  greater  than  30  feet  between  the  consecutive  cracks,  a sin- 
gle pass  would  suffice.  However,  if  the  frequency  of  transverse  cracks  is,  say,  N cracks  every  30 
feet,  it  would  take  N passes  over  the  highway  to  seal  them  up. 

Possible  Payoffs  from  the  Proposed  Research 

In  the  following  we  describe  a proposed  roadway  inspection  system.  If  the  objectives  could 
be  accomplished,  the  following  benefits  would  be  realized: 

□ Roadway  defects  such  as  cracks  would  be  detected  by  our  system,  with  perfor- 
mance better  than  that  of  human  inspectors.  The  minimum  crack  size  would 
be  less,  meaning  that  preventive  maintenance  could  be  earned  out  earlier, 

1 Of  course,  the  precise  time  required  to  fill  up  a crack  will  also  depend  on  the  width  and  the 
depth  of  the  crack;  both  of  these  parameters  can  be  estimated  with  a range  sensor.  We  will  assume 
that  it  is  possible  to  regulate  the  flow  of  the  sealant  to  adapt  to  the  width  and  the  depth  of  the 
crack. 
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FIG.  A:  Concept  drawing  of  a planar  robotic  arm  for  sealing  longitudinal  and  transverse  pavement 
cracks.  S£e  text  for  details. 
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leading  to  increased  savings  in  repairs. 

□ Quantitative  measures  of  performance  - detection  rate,  false-alarm  rate,  etc.  - 
could  be  accurately  characterized.  This  is  not  currently  possible  with  the 
highly  subjective  human  decision-making  process. 

□ As  the  inspection  (and,  for  defects  of  limited  severity,  the  repair)  could  be 
done  at  highway  speeds,  much  more  roadway  than  the  currently  limited  sample 
could  be  inspected.  What  is  more,  the  repair  would  be  carried  out  simultane- 
ously. Inspection  and  repair  would  become  a one-pass  operation. 

Discussion 

Road  surface  inspection  is  a boring,  repetitive  task  consisting  of  inspection  and  quantitative 
evaluation  of  a manufactured  surface.  As  such,  it  is  an  ideal  application  of  machine  vision.  In 
the  following  we  propose  methods  for  the  automatic  detection  of  roadway  defects,  and,  where 
possible,  their  automatic  repair. 

Human  experts  recognize  many  distinct  forms  of  road  surface  damage  or  deterioration; 
[GADOT]  presents  one  of  many  classification  schemes.  However,  all  forms  are  presently 
detected  by  human  inspectors  who  simply  look  at  the  road.  This  means  that  photometric  and 
range  measurements  would  possess  adequate  information  for  the  detection  of  all  currently  recog- 
nized problems. 

Figure  1 shows  a cross-section  of  a road  with  some  commonly  recognized  problems 
labeled.  Since  a pothole  is  the  largest  such  structure,  it  is  easy  to  detect,  the  aberrant  profile 
exhibited  in  the  range  data  clearly  indicates  its  presence  and  accurately  defines  its  boundaries. 
The  exposed  aggregate  would  be  of  a different  color,  and  so  a pothole  would  also  be  easily 
detected  in  photometric  imagery. 

We  have  turned  our  attention  to  a more  difficult  inspection  problem,  crack  detection  and 
repair.  While  cracks  can  grow  quite  wide,  thus  becoming  easy  to  detect,  it  is  best  if  they  are 
detected  early  so  that  prompt  repair  can  prevent  further  damage.  This  requires  the  detection  of 
quite  small  cracks,  which  means  that  vision  data  must  be  gathered  at  a fine  scale,  so  that  features 
only  0.1"  wide  or  even  narrower  be  detected.  Since  cracks  may  be  transverse  or  longitudinal,  or 
at  any  angle  in  between,  it  is  necessary  to  gather  data  at  similar  sampling  rates  both  across  the 
road  and  along  the  direction  of  travel.  The  requirements  for  storing  the  amount  of  data  associ- 
ated with  even  a short  segment  of  a road  would  be  astronomical.  The  only  solution  is  a system 
that  analyzes  the  road  as  it  travels  over  it,  storing  only  the  information  needed  to  describe 
detected  problems. 

What  sort  of  information  would  be  useful  for  making  this  type  of  decision?  We  feel  that 
both  range  and  reflectance  information  should  be  used.  Consider  the  highly  schematic  system 
shown  in  Figure  2.  Imagine  an  array  of  balls,  or  wheels,  of  differing  diameters,  free  to  roll  along 
the  road  and  follow  its  contour.  If  there  were  a way  to  record  the  up-and-down  motions  of  these 
individual  sensors,  the  result  would  be  a 3-D  map  of  the  road  surface  showing  its  texture  and 
shape  at  varying  scales.  The  larger  balls  or  wheels  would  describe  the  overall  shape  of  the  road, 
and  would  be  useful  for  the  detection  of  such  relatively  large-scale  phenomena  as  rutting.  The 
smaller  sensors  would  measure  the  surface  at  a finer  scale,  detecting  narrow  cracks. 

We  took  a sample  of  slightly  deteriorated  road  surface,  and  used  a robotic  manipulator  with 
a force-torque  sensor  to  drag  a probe  over  the  sample  surface,  attempting  to  maintain  constant 
force/torque  conditions.  Records  the  paths  of  the  robot  arm  as  it  moved  the  probe  in  a grid 
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pattem  over  the  sample  yielded  measurements  of  a form  similar  to  those  of  our  hypothetical  sys- 
tem of  Figure  2.  Some  of  this  data  is  plotted  in  Figures  3 and  4. 

The  problem  is  that  electro-mechanical  recording  devices  are  too  slow  to  gather  data  of  the 
desired  scale  while  the  platform  moves  at  a realistic  speed.  For  this  reason  we  suggest  using 
range  sensors  of  the  general  types  commonly  used  in  machine  vision.  Figure  5 shows  a 
structured-light  range  sensor  - a system  in  which  a projector  casts  a distinctive  pattern  on  a 
scene  and  a camera  views  the  scene  from  an  offset  position.  Knowledge  of  the  optics  and 
geometry  of  the  system  allow  the  (row, column)  information  gathered  by  the  camera  to  be  con- 
verted into  (x,y,z)  measurements,  building  a range  map  of  the  road  as  the  system  travels  over  it. 
The  range  resolution  characteristics  are  a function  of  the  sensor  design,  it  would  need  to  be 
designed  so  that  the  structures  of  interest  can  be  seen. 

How,  then,  would  the  analysis  be  done?  We  suggest  the  application  of  morphological 
operators  to  the  range  map,  among  other  techniques.  Figure  6 shows  the  z component  of  the 
(x,y,z)  measurements  for  a sample  of  deteriorating  asphalt,  with  greater  values  of  z represented 
by  brighter  pixels.  The  crack  network  leading  from  the  left  to  the  bottom,  via  the  center,  is  very 
noticeable.  Figure  7 shows  a similar  image  for  two  samples  of  concrete,  one  of  which  is 
cracked.  Again,  the  flaw  is  strikingly  obvious.  Morphological  operators,  which  can  be 
mathematical  models  of  rolling  balls,  make  such  features  more  striking.  Figure  8 shows  the 
application  of  such  an  operator  to  the  image  of  z as  intensity  for  the  concrete  samples.  Other 
than  the  ends  of  the  samples,  the  only  noticeable  feature  is  the  crack. 

For  details  that  could  be  seen  in  simple  photometric  imagery,  a system  similar  to  that  of 
Figure  9 would  suffice.  A camera  with  multiple  light  sources  would  view  the  road  as  the  system 
was  transported.  Again,  no  attempt  should  be  made  to  store  the  imagery.  It  should  instead  be 
interpreted  as  it  is  gathered,  and  only  symbolic  information  about  defects  and  their  locations 
retained. 

Figure  10  shows  what  is  called  registered  intensity  data  for  the  asphalt  sample  - it  is  the 
intensity  measurements  made  at  the  same  locations  for  which  (x,y,z)  values  were  measured,  with 
z displayed  in  Figure  6.  Similarly,  Figure  11  shows  registered  intensity  data  for  the  concrete 
samples.  Morphological  operators  can  also  be  applied  to  the  photometric  data,  as  shown  in  Fig- 
ure 12. 

Our  proposed  system  would  thus  make  a large  number  of  measurements  of  each  visible 
segment  of  the  road  surface.  What  should  be  done  with  these  multiple  channels  of  information? 
We  feel  that  an  architecture  similar  to  that  shown  in  Figure  13  might  produce  optimal  results. 
Morphological  operators  of  differing  characteristics  would  be  applied  to  both  range  and  pho- 
tometric imagery,  and  the  outputs  applied  as  the  input  to  neural  networks. 

Once  the  cracks  had  been  detected  in  imagery,  the  information  about  the  precise  location  of 
the  sensor  platform  at  the  time  the  data  was  gathered  would  be  used  to  drive  machinery  appropri- 
ate for  sealing  that  type  of  surface.  Our  experience  with  mobile  robotics  indicates  that  it  would 
be  a mistake  to  rely  on  odometry  to  coordinate  actions  across  several  vehicles.  If  the  sealing  is 
done  by  a second  vehicle,  then  some  sort  of  accurate  relative  positioning  method  is  needed  to 
accurately  drive  a “blind”  repair  arm.  We  feel  that  the  best  solution  would  be  to  have  the  repair 
done  from  the  same  platform,  so  a rigid  frame  of  reference  between  sensor  and  actuator  is  main- 
tained. Some  form  of  visual  servoing  might  be  needed  to  drive  the  end  effector  along  the  crack. 
That  task  is  similar  to  the  seam-tracking  done  by  existing  welding  robots,  and  so  a mature  tech- 
nology is  available  for  application.  Given  that  the  robotic  manipulators  will  be  mounted  on  a 
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moving  platform,  some  work  is  obviously  needed  in  the  area  of  dynamic  path-planning  and  in 
the  coordination  of  the  positioning  of  the  end  effector  with  respect  to  the  motion  of  the  vehicle. 

Performance 

An  important  question  relevant  to  this  proposal  is:  How  fast  can  such  inspection  and  repair 
be  carried  out  simultaneously?  In  other  words,  what  will  be  the  speed  of  the  vehicle  on  which 
this  inspection  and  repair  system  will  be  mounted?  If  inspection  alone  were  our  objective,  we 
believe  that  could  be  done  at  highway  speeds  using  a bank  of  laser  beams,  each  scanning  a small 
area  of  the  road  surface;  in  other  words,  the  entire  inspection  process  could  be  carried  out  in  a 
highly  parallel  fashion  using  a bank  of  rather  inexpensive  microprocessors,  each  microprocessor 
processing  the  output  of  a single  laser  beam  and  making  local  decisions  concerning  the  state  of 
the  road  surface.  However,  with  the  added  objective  of  real-time  repair,  the  overall  speed  will 
be  limited  by  the  kinematics  and  dynamics  of  the  robot  arm  sealing  the  cracks.  While  it  would 
be  relatively  easy  to  fill  the  longitudinal  cracks,  for  filling  the  transverse  cracks  the  driver  of  the 
maintenance  vehicle  will  have  to  relinquish  the  speed  control  to  the  robot  arm.  In  other  words, 
the  sensors  mounted  on  the  robot  arm  will  decide  how  fast  the  arm  will  be  able  to  move  to  fill  a 
transverse  crack  and  will  automatically  adjust  the  speed  of  the  vehicle  accordingly.  From  our 
preliminary  calculations  at  this  time,  we  believe  that  it  would  be  possible  to  design  a prototype 
planar  arm,  with  two  degrees  of  freedom,  that  would  allow  a vehicle  to  move  at  approximately 
10  miles  per  hour  while  crack  filling  operation  is  going  on. 

Deliverables 

This  research  effort  will  progress  along  two  parallel  tracks  during  the  first  two  years  and 
then,  in  the  third  year,  the  two  tracks  will  be  merged.  One  track  will  deal  with  the  development 
of  2D  and  3D  vision  algorithms  for  robust  detection,  delineation,  and  characterization  of  cracks. 
The  second  track  will  focus  on  the  development  of  the  special  arm  needed  for  the  task.  Initially, 
the  arm  will  be  attached  to  a translational  motor  in  the  laboratory  to  simulate  vechile  motion. 
This  setup  will  be  used  fine  tune  the  ability  of  the  arm  to  track  what  will  be  cracks  simulated 
with  the  help  of  synthesized  patterns  on  the  lab  floor.  In  the  last  year,  the  vision  algorithms  will 
be  installed  in  the  controller  of  the  arm  and  entire  system  field  tested.  The  main  deliverable  of 
this  research  effort  will  be  the  new  robotic  arm,  the  controller  for  the  arm,  and  the  sensory  sys- 
tem. 

Budget  Justification 

Support  is  being  requested  for  10%  of  time  for  the  PI,  25%  in  summer,  and  three  graduate 
research  assistants.  For  the  engineering  effort  needed  to  fabricate  the  arm  and  the  controller  for 
the  aim,  the  budget  includes  a full-time  engineer,  a full-time  technician/mchinist,  and  a full-time 
software/control  engineer.  While  two  of  the  graduate  assitants  will  spend  their  time  on  the 
software  aspects  of  the  program,  one  will  focus  on  the  research  aspects  of  the  hardware  issues. 
We  believe  that  the  parts  for  the  planar  robotic  arm  and  the  associated  controller  will  cost 
roughly  $40,000.  The  cost  of  the  sensory  system  and  a processor  for  the  2D  and  3D  information 
would  cost  $38,500.  for  building  a shrouded  structured-light  sensing  system  that  would  work  in 
an  outdoor  environment  This  system  will  include  a high-speed  image  processing  system  for 
grabbing  multiple  video  frames  and  for  video-rate  low-level  processing  of  these  frames  (cost: 
$33,000),  two  high-quality  CCD  color  cameras  (cost  $3,000),  multiple  laser  sources  (cost: 
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$2,000),  various  and  sundry  components  for  mounting  the  lasers  and  cameras  (cost  $500). 
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Introduction 

Recent  studies  carried  out  by  a group  convened  by  FHWA  and  NIST  [FHWA]  indicated 
that  the  automation  of  bridge  deck  construction  would  be  highly  beneficial.  We  believe  that 
such  automation  could  enhance  productivity  and  efficiency  by  at  least  30%.  In  this  proposal,  we 
will  outline  the  steps  that  are  ripe  for  automation.  Given  the  diverse  nature  of  the  tasks  involved, 
we  believe  that  a consortium  approach  would  be  best  for  this  project,  hence  the  reason  for  the 
organizations  and  the  principal  investigators  listed  above.  This  consortium  will  work  on  the 
various  aspects  of  an  optimal  automated  screed.  Purdue  will  address  the  sensory  and  mechanical 
problems  associated  with  the  flow  and  leveling  of  the  concrete  and  with  the  inspection  of  the 
subdeck  concrete.  RPI,  in  conjunction  with  NCSU,  will  work  on  the  automation  of  rebar  han- 
dling and  assembly.  NIST  will  work  on  the  higher-level  integration  aspects  of  the  component 
technologies  that  will  emerge  from  Purdue,  RPI,  and  NCSU. 

The  SCREED  Steps  That  Can  Be  Automated 

[1]  Inspection  of  the  sub-grade  to  ensure  all  codes  are  met,  including: 

(a)  Adequate  pads  and  spacers  between  beams  and  sub-deck  panels. 

(b)  Proper  alignment  of  sub-deck  panels. 

(c)  Absence  of  cracks  or  voids  in  sub-deck  panels. 
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[2]  Laying  of  rebar,  including  placing  the  chairs  and  tying  the  rebar. 

[3]  Application  of  water  and/or  other  agents  as  needed. 

[4]  Monitoring  the  flow  of  supplied  concrete  to  ensure  proper  volume. 

[5]  Applying  mechanical  vibration  as  needed  to  ensure  proper  material  flow. 

[6]  Monitoring  the  supplied  concrete  to  ensure  that  standards  regarding  density  and  plasticity 
are  continuously  met. 

[7]  Monitoring  the  constructed  deck  extruded  behind  the  machinery,  to  ensure  internal  and  sur- 
face characteristics  meet  all  standards. 

It  is  possible  that  other  operations,  such  as  floating  or  tining,  could  also  be  included  in  the  final  system 
design. 

Addressing  these  points  in  the  order  of  operation,  we  see  that  inspection  is  the  first  step.  This 
would  most  likely  be  visual  inspection,  and  can  be  implemented  as  a combination  of  2-D  and  3-D  visual 
sensing.1  Cameras,  possibly  used  in  conjunction  with  structured- light  sources,  could  be  mounted  on  a 
gantry  over  the  front  of  the  machine.  These  sensors  could  ensure  that  the  sub-deck  panels  are  correctly 
aligned  with  the  beams,  that  they  are  not  cracked,  etc.  In  the  highly  schematic  Figure  1,  the  right-most 
gantry  is  shown  as  carrying  out  this  task. 

Mechanical  arms  on  the  front  of  the  machine  will  be  used  to  pluck  rebar  from  a supply  bin  and 
place  them  on  chairs,  properly  aligned.  After  the  rebar  is  tied,  further  visual  inspection  will  ensure  its 
proper  placement.  Rebar  assembly  and  tying  would  be  done  by  the  second  of  the  linked  gantries  in  Fig- 
ure 1. 

Water  must  be  sprayed  before  concrete  is  poured,  to  avoid  drawing  too  much  water  out  of  the  mix. 
Visual  inspection  could  ensure  that  all  surfaces  had  been  adequately  wetted,  driving  the  spray  mechan- 
ism to  correct  any  areas  slighted. 

We  propose  to  use  simple  range  sensors  to  determine  the  cross-section  of  the  sub-decking  as  the 
screed  progresses,  and  also  to  measure  the  cross-section  of  the  supplied  concrete.  In  Figure  1, 
structured-light  projectors  on  the  screed,  coupled  with  cameras  mounted  on  the  second  gantry,  make 
these  measurements.  In  this  way  the  screed  could  ensure  adequate  supply,  but  no  more,  by  communi- 
cating with  the  concrete  supply  devices.  We  believe  that  these  material  sources,  probably  concrete 
pumps  in  this  case,  can  be  linked  into  a site-wide  packet  radio  network.  A packet  cluster  architecture 
would  allow  random  inter-machine  communication  between  any  two  pieces  of  equipment  on  a site,  as 
well  as  central  control  by  a machine  supplying  a primary  service.  We  have  not  attempted  to  show  the 
actual  concrete  supply  in  Figure  1,  since  we  feel  that  a flexible  approach  is  required.  The  automated 
system  we  propose  would  communicate  with  whatever  material  supply  systems  were  available,  be  that  a 
conveyor  system,  a concrete  pump,  or  whatever,  to  request  concrete  (and  rebar)  as  needed. 

Finally,  as  the  integrated  system  progresses  along  the  bridge,  2-D  and  3-D  sensors  could  inspect 
the  finished  product.  The  cross-sectional  profile  could  be  measured  to  a high  degree  of  accuracy  in 
order  to  ensure  that  all  standards  were  being  met.  These  measurements  could  form  a record  that  would 
be  available  for  later  analysis.  Also,  the  surface  moisture  characteristics  could  be  monitored,  and 


1 While  ultrasonics  is  used  for  detection  of  internal  stresses  and  micro-cracking,  it  is  perhaps 
ill-suited  for  this  application. 
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corrected  as  necessary. 

Automated  Rebar  Handling  And  Assembly 

A major  component  of  bridge  deck  construction  consists  of  the  handling,  laying  and  tying  of  rebar 
to  provide  the  reinforcement  for  the  concrete  deck.  In  current  practice,  rebar  bundles  are  delivered  to  a 
job  site  and  stacked  for  future  use.  Bundles  are  manually  sorted  and  arranged  at  a staging  site,  tran- 
sported to  the  bridge  deck  at  the  time  of  construction,  and  individual  pieces  are  again  handled  manually 
for  positioning  and  tying.  Automation  of  these  procedures  would  result  in  more  efficient  handling  of 
materials,  higher  quality  in  the  positioning  and  tying  of  the  reinforcement,  increased  safety  due  to  the 
reduced  manual  handling  of  heavy  parts,  and  reduction  in  the  incidence  of  back  injuries  resulting  from 
bending  and  tying  of  the  rebar. 

Two  major  steps  in  this  process  may  be  addressed  by  automation  techniques  in  order  to  achieve 
these  goals: 

( 1 ) .  Computer-aided  design,  organization  and  tracking  of  rebar  materials: 

Rebar  pieces  should  be  bundled,  delivered,  and  handled  in  lots  which  map  directly  onto  the  deck  assem- 
bly process  itself.  Computer-aided  design  tools  should  be  used  to  specify  the  rebar  design  and  layout. 
The  designer  or  construction  process  planner  then  specifies  the  order  or  layout  and  assembly  of  the  rebar 
structure.  Based  on  this  order  of  assembly,  the  rebar  pieces  are  specified  as  groups  or  kits,  and  this 
grouping  is  specified  to  the  rebar  manufacturer  for  delivery  in  the  prescribed  form  with  labels  attached 
to  the  groups.  This  planning  and  labeling  avoids  rehandling  and  sorting  at  the  construction  site.  Ideally, 
the  delivery  of  the  materials  themselves  would  be  coordinated  with  the  construction  schedule  so  that  the 
rebar  materials  are  delivered  directly  for  layout  and  assembly  of  the  deck  and  do  not  require  separate 
staging.  This  delivery  and  assembly  schedule  is  worked  into  the  overall  project  plan  to  very  the  coordi- 
nation of  rebar  assembly  with  deck  pouting  in  the  integrated  deck  construction  system.  Efficiency  in 
transport,  handling,  and  assembly  is  gained  by  careful  planning  and  scheduling  of  the  tasks. 

(2) .  Automated  assembly  and  tying  of  the  rebar  parts: 

The  assembly  process  itself  will  be  integrated  into  the  automated  decking  system  which  travels  along 
the  bridge  and  performs  the  tasks  of  rebar  tying  concrete  pouring,  smoothing  and  sensing  in  a single 
pass.  The  automated  handling  and  tying  of  the  rebar  structure  is  a particularly  challenging  phase  of  this 
process.  It  is  unlikely  that  general  robot  arms  would  effectively  accomplish  this  task  due  to  the  length, 
flexibility,  and  weight  of  the  rebar  parts  themselves.  A more  realistic  scenario  for  rebar  assembly  is 
shown  schematically  in  Figure  (2),  where  manually  assisted  laying  of  the  longitudinal  rebar  prepares  the 
way  for  automated  laying  and  tying  of  the  transverse  bars.  The  longitudinal  bars  are  fed  directly  from  a 
delivery  vehicle  or  positioning  in  kits  using  a conventional  crane.  The  manual  operation  sets  the  longitu- 
dinal bars  on  supports  in  their  approximate  final  position.  The  first  set  of  supports  have  built  in  physical 
spacers  which  align  the  ends  of  the  bars  with  the  prior  set.  The  bar  spacing  for  the  remaining  length  will 
be  set  by  the  assembly  and  tying  apparatus  which  tracks  the  longitudinal  bars  and  adjusts  their  positions 
as  it  ties. 

The  transverse  rebar  placement  and  tying  machine  travels  with  the  integrated  decking  system  and 
is  aligned  to  physical  or  optical  guides  along  both  sides  of  the  bridge.  The  machine  contains  a feeder  bin 
of  transverse  bars  which  have  been  loaded  directly  be  crane  according  to  the  labelled  groupings.  Each 
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transverse  bar  is  fed  and  tied  as  the  machine  proceeds  along  the  longitudinal  bars.  The  longitudinal  bars 
are  tracked  and  positioning  prior  to  tying.  The  proposed  tying  mechanism  itself  is  a plastic  "tie-wrap" 
loop  analogous  to  ties  which  are  used  to  secure  cables  and  wires.  Such  a tie-wrap  loop  tightens  securely 
and  requires  much  less  complexity  in  the  automated  Wing  mechanism.  The  intersections  of  each 
transverse  bar  with  all  the  longitudinal  bars  are  tied  simultaneously  by  a set  of  tie-wrap  mechanisms. 
The  details  of  the  feeding,  tracking,  and  tying  mechanisms  are  not  provided  in  this  description,  but  have 
been  developed  to  a stage  of  conceptual  design.  Specific  attention  is  given  to  the  need  to  hold  and  tie  the 
transverse  bars  while  the  machine  is  in  motion.  An  added  feature  of  the  integrated  machine  could 
include  an  inspection  device  to  scan  the  resulting  rebar  mesh  and  confirm  both  correct  geometry  and 
placement  of  ties  and  supports. 

The  automated  rebar  installation  system  described  here  represents  several  innovations  in  the  pro- 
cedures and  the  mechanisms  available  for  automated  bridge  deck  construction.  The  impact  of  these 
innovations  would  occur  as  a result  of  both  of  these  processes.  Improvement  in  design  and  planning 
tools  provide  a direct  link  into  the  organization  and  handling  of  the  rebar  materials.  This  change  would 
reduce  the  time  required  to  handle  the  rebar  and  increase  the  safety  of  the  workers.  Eliminating  the  stag- 
ing and  sorting  of  the  rebar  materials  could  improve  the  efficiency  of  this  phase  of  the  project  by  as 
much  as  50%. 

The  automated  rebar  laying  and  tying  system  would  perform  the  rebar  installation  as  a parallel 
operation  with  the  concrete  pouring.  While  the  longitudinal  layout  would  still  be  done  largely  with 
manual  methods,  the  feeding  and  organization  of  the  materials  would  be  improved.  The  automated 
transverse  bar  assembly  and  tying  would  be  entirely  automated.  Only  the  rebar  feeder  would  need  to  be 
refilled  as  the  task  proceeded.  This  reduction  in  work,  force  and  time  required  would  increase  the 
efficiency  of  this  phase  of  the  project  by  at  least  50%  and  have  additional  benefits  on  health  and  safety 
issues. 

The  approach  to  rebar  handling  and  assembly  described  here  for  bridge  decking  will  impact  other 
rebar  assembly  processes  in  road  construction.  Both  the  improved  design  and  tracking  approached,  and 
the  automated  laying  and  tying  devices  can  be  adopted  to  other  problems  including  center  and  sidewall 
construction  and  pilings.  While  the  uniformity  of  the  bridge  deck  problem  suggests  that  it  is  particularly 
well-suited  to  automated  technologies,  these  extensions  to  related  areas  will  have  similar  benefits. 

Performance  Issues 

What  will  be  the  productivity  enhancement  if  such  a system  is  deployed?  A precise  answer  to  that 
will  depend  on  how  many  robotic  arms  and  sensors  are  mounted  on  such  a system.  [On  a wide  span,  like 
the  one  shown  to  us  in  Los  Angeles,  separate  robot  and  sensor  suites  could  be  assigned  to  each  12  foot 
section  of  the  deck.]  Although  admittedly  a very  rough  guess  at  this  time,  even  without  any  parallelism 
in  robot  arms  and  sensors,  we  believe  that  the  proposed  system  would  increase  the  productivity  by  at 
least  30%.  Moreover,  the  quality  of  the  decking  would  be  superior,  more  consistent,  and  less  prone  to 
human  error.  This  30%  figure  is  based  on  the  manual  observation  of  the  current  practice  of  deck  con- 
struction in  California.  We  observed  the  different  cycles  of  interlocking  steps  that  go  into  the  process  of 
laying  down  a deck  and  how,  more  often  than  not,  the  personnel  assigned  to  one  step  have  to  stand 
around  and  wait  while  the  other  related  steps  are  brought  to  completion. 

Deliverables 

Our  plan  is  to  carry  out  incremental  automation  of  an  existing  machine  for  laying  out  bridge  decks. 
This  we  believe  will  be  the  least-cost  approach  to  gaining  insights  into  the  design  of  the  various  sensor 
systems  that  will  be  needed  for  the  different  automation  steps.  During  the  first  year,  we  will  develop  the 


I 


-5- 


sensor  system  for  monitoring  the  quality  of  the  concrete  deck  laid  out  by  the  existing  machine.  The 
quality  measures  would  be  the  uniformity  of  the  height  of  the  deck  and  of  the  uniformity  of  the  texture 
of  the  surface.  Both  of  these  quality  measures  are  evidendy  important  and,  in  later  phases  of  research, 
would  be  used  to  initiate  remedial  measures,  such  as  the  screed  going  again  over  the  affected  area  after 
the  concrete  delivery  system  has  either  poured  additional  concrete  over  the  area  or  scraped  some  of  it 
off. 

First  year’s  effort  will  also  result  in  the  computerization  of  the  design  of  rebar  layouts  and  a system 
for  the  tagging  of  rebars  so  that,  in  the  future,  it  would  be  possible  to  design  a “rebar  feeder"  that  would 
automatically  layout  the  bars.  [Initially,  this  system  of  the  future  would  require  human  assistance  with 
the  longitudinal  bars.]  The  detailed  mechanical  design  of  an  actual  machine  for  laying  out  the  rebars 
and  tying  them  would  require  large  engineering  support  and  is  not  incluned  in  this  proposal.]  Our  first 
year’s  effort  would  only  involve  analyzing  the  feasibility  of  such  a machine  and  testing  the  feasibility 
with  computer  simulations. 

In  the  second  year,  we  will  develop  the  system  for  monitoring  the  volume  of  the  poured  concrete 
before  it  is  leveled  by  the  screed.  For  quality  control,  it  is  important  that  the  volume  of  the  concrete, 
poured  initially  into  small  heaps,  is  as  tightly  as  possible  in  proportion  to  what  is  needed  to  fill  up  a cer- 
tain area  of  the  deck.  Any  large  variation  in  poured  concrete,  especially  when  the  amount  poured  is  on 
the  deficit  side,  manifests  itself  in  the  form  of  depressions  that,  with  the  current  technology,  have  to  be 
gone  over  manually  and  resmoothed  by  the  motions  of  the  screed.  The  development  of  the  sensor  sys- 
tem for  monitoring  the  volume  of  poured  concrete  is  expected  to  eliminate  this  manual  intervention. 

Another  planned  goal  for  the  second  year  deals  with  analyzing,  from  kinematic  and  controls  stand- 
points, the  various  alternatives  for  the  rebar  manipulation  system.  As  a result  of  this  study,  we  will  be 
able  to  make  definitive  statements  about  minumum  number  of  degrees  of  freedom  that  would  be  needed 
for  laying  down  rebars.  Of  course,  these  conclusions  would  depend  on  assumed  models  for  both  how 
the  bars  are  packaged  and  how  the  packages  are  delivered  to  the  bridge  deck. 

The  third  year  will  focuss  on  the  issues  of  integration  of  the  component  technologies  developed  in 
the  first  two  years.  This  integration  will  allow  us  to  simultaneously  control  the  flow  of  concrete  and  do 
so  in  such  a manner  that  the  quality  of  the  deck,  in  terms  of  the  uniformity  of  thickness  and  texture  of 
the  surface,  is  ensured. 

Budget  Justification 

The  Purdue  part  of  the  budget  includes  10%  time  for  the  PI,  25%  in  summer,  and  two  graduate  assis- 
tants. The  budgets  for  RPI  and  NCSU  have  been  included  as  subcontracts  in  the  Purdue  budget  Purdue 
has  a policy  of  charging  its  overhead  on  the  first  $25,000  of  a subcontract.  The  breakdown  for  the  RPI 
budget  is  available  and  is  as  follows: 

A Sanderson  (PI  @ 10%  + Summer) = $19,558 


GRA' s (2)  (AY  Stipend/ Summer) = 32,776 
GRA' s (Tuition)  = 24,422 
Secretarial  Assistance  = 2,355 

Fringe  Benefits  = 6,771 
Travel  = 2,500 
material,  Supplies&Services  = 2,500 
Indirect  Costs  (OH)  = 36,553 
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Total  =$127,435 

The  budget  also  includes  a first-year  request  of  $43,500  for  building  a shrouded  structured-light  sensing 
system  that  would  work  in  an  outdoor  environment  This  system  will  include  a high-speed  image  pro- 
cessing system  for  grabbing  multiple  video  frames  and  for  video-rate  low-level  processing  of  these 
frames  (cost:  $33,000),  two  high-quality  CCD  color  cameras  (cost  $3,000),  multiple  laser  sources  (cost: 
$2,000),  two  high-precision  ball  screws  for  the  scanning  machanism  ($5,000),  various  and  sundry  com- 
ponents for  mounting  the  lasers  and  cameras  (cost  $500). 
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1.  BACKGROUND  AND  MOTIVATION 


1. 1 Background  and  Discussion  of  Related  Research 

Almost  all  production  oriented  robots  today  are  utilized  within  the  manu- 
facturing industry.  One  main  reason  why  true  robots  are  hardly  seen  in  the  con- 
struction environment  is  because  of  the  existing  unstructured  and  complex 
conditions.  In  addition,  construction  mostly  takes  place  in  an  uncontrolled 
environment,  exposed  to  the  weather,  dust  and  noise.  The  attributes  of  the 
materials  to  be  handled  range  from  large,  heavy,  bulky,  and  non-homogeneous  to 
light,  fragile,  and  homogeneous. 

Despite  the  many  difficulties,  opportunities  for  applying  high  technology  in 
construction  are  abundant.  For  certain  applications  and  situations,  such  as 
construction  in  hazardous  areas  (i.e.,  nuclear  waste  disposal,  space  construction), 
robotic  technology  is  unavoidable  [1].  However,  applying  high  technology  in  these 
areas  requires  empirical  as  well  as  theoretical  research. 

Excavation  of  soil  and  rock  is  a high  volume  and  repetitive  construction 
operation.  Some  30%  of  the  earth's  crust  is  shale  or  mudstone,  much  of  which  can 
be  excavated  by  using  a backhoe,  front-end  loader,  or  other  heavy  mechanical 
excavator.  Because  of  their  versatility,  backhoe  excavators  are  especially  popular  on 
construction  sites.  It  is  estimated  that  a fully  automated  excavating  machine 
performing  at  25%  the  efficiency  of  an  expert  human  operator  would  be 
commercially  feasible  [2], 

Studies  on  the  applications  of  robotic  excavation  and  robotic  excavators  have 
been  undertaken  by  several  researchers  worldwide.  Previous  research  results  can  be 
classified  into  three  categories.  The  first  category  consists  of  work  on  geometric 
planning  for  robotic  excavation.  A few  researchers  have  worked  on  gross 
specifications  of  digging  [2,  3].  This  type  of  planning  abstracts  the  world  into  a 
geometric  basis  and  does  not  take  into  account  considerations  of  mass,  force  or  any 
soil  property.  The  entire  excavation  task  is  segmented  into  a sequence  of  geometrical 
shapes  before  excavation  begins.  The  second  category  consists  of  work  on  controlling 
the  robotic  excavator  along  a planned  trajectory.  Here,  different  control  principles 
and  sensing  technologies  are  proposed.  Vaha  studied  and  established  a kinematic 
and  dynamic  control  model  for  a robotic  excavator  [4].  Bullock  and  Oppenheim 
developed  an  approach  for  force-cognitive  robotic  excavation  [5].  Tochizawa  et  al. 
reported  about  an  automated  excavator  for  excavating  a trench  for  drainage  using 
laser  guidance  [6].  The  authors  showed  that  the  laser  guidance  helped  to  decrease  the 
labor  hours  and  increase  the  digging  accuracy.  Finally,  there  has  also  been  some 
interest  in  remote  controlled  excavators  (or  teleoperated  excavators)  for  construction 
and  hazardous  waste  handling.  The  aim  of  this  work  is  to  remove  the  human 
operator  from  the  immediate  work  site.  Langreth  reported  about  an  advanced 
teleoperated  hydraulic  excavator  [7].  This  excavator  operates  with  a master-slave 
control  and  incorporates  the  force-feedback  control  so  that  the  human  operator  can 
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not  only  control  the  mechanical  arm  very  easily,  but  also  "feel"  the  obstacles  that  the 
bucket  hits.  Provided  with  video  images  from  three  cameras,  the  operator  sits 
remotely  inside  a building  or  a vehicle,  manipulating  and  observing  the  operations. 

1.2  The  Hazards  of  Trenching  and  Pipe-Laying 

The  opening  of  trenches  to  bury  sewer,  water,  and  other  utilities  is  an  operation 
which  can  be  observed  on  almost  any  highway  construction  project.  Traditionally,  a 
surveying  crew  will  first  stake  out  and  mark  the  location  of  a planned  trench.  Later, 
a construction  crew  will  begin  excavating  the  trench  to  the  desired  depth  using 
traditional  surveying  equipment  and  approximation  to  control  depth  and  direction. 
Sometimes  the  stakes  have  been  accidentally  removed  or  damaged  by  the  time  the 
construction  crew  starts.  Traditional  excavation  requires  persons  to  enter  the  trench 
with  shovels  to  even  out  the  bottom,  to  lay  pipe,  etc. 

The  hazardous  nature  of  construction  work,  especially  that  related  to  excavation 
work,  is  well  documented.  The  fatality  rate  was  estimated  by  OSHA  at  50.8  deaths 
per  100,000  workers  per  year  for  1984-1988,  whereas  for  construction  work  generally, 
it  was  estimated  at  24.8  deaths  per  100,000  employees.  Similarly,  trenching  cave-in 
fatalities  have  been  estimated  by  NIOSH  at  75  per  year,  and  lost  workday  injuries 
due  to  cave-ins  at  1000  per  year.  In  a recent  report  prepared  by  NIOSH,  based  on 
OSHA's  inspection  data,  it  was  estimated  that  at  least  another  97  persons  were  killed 
as  a injury  among  construction  workers,  including  those  doing  excavation  work,  is 
about  two  time  the  all  industry  average  (i.e.,  15.1  injuries  per  100  workers  in 
construction  compared  with  7.7  injuries  per  100  workers  in  all  industries)  [8]. 

The  major  occupational  hazards  of  excavation  work  result  from  cave-ins,  from 
exposure  to  underground  utilities,  and  from  material  or  equipment  falling  into  the 
open  trench.  Precautions  against  cave-ins  include  bracing,  sloping,  benching,  use  of 
shields,  and  freezing.  However  the  proper  use  of  these  techniques  requires  an 
understanding  of  the  importance  of  such  factors  as  excavation  depth  and  width,  soil 
type,  hydraulic  pressure,  and  other  specific  conditions  present  at  the  worksite. 

1. 3 Scenario  of  Laser  Guided  Robotic  Excavation 

Providing  a technology  which  would  eliminate  or  sharply  reduce  the  time  a 
worker  has  to  spend  in  the  trench  would  be  of  great  significance  to  construction 
safety.  This  research  project  proposes  to  use  the  available  laser  guidance  technology 
to  provide  a means  to  automate  trench  excavation.  Figure  1 depicts  an  overview  of 
the  trench  digging  operation  using  laser  guidance  with  a real  excavator. 
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Figure  1.  Proposed  Trench  Excavation  Using  Laser  Guidance 


The  proposed  research  project  would  investigate  the  effectiveness  of  the  laser 
guidance  positioning  approach  in  assisting  the  trenching  operation  by  eliminating: 
1)  surveying  before  and  during  the  operation,  2)  risky  exposure  of  workers  in  the 
trench  during  the  operation,  and  3)  labor  intensive  hand  shoveling  at  the  bottom  of 
the  trench. 


1. 4 Scenario  of  Laser  Guided  Robotic  Pipe  Laying 

The  proposed  experimental  work  on  robotic  pipe  laying  is  another  application 
for  laser  guidance  technology  in  the  construction  area.  A conceptual  overview  of  the 
envisioned  future  of  laser  guided  pipe  laying  is  shown  in  Figure  2.  The  final  goal 
here  is  to  improve  the  accuracy  and  efficiency  of  pipe  laying.  Laser  beams  and  laser 
planes  will  be  used  as  the  calibration  and  measurement  tools. 
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Figure  2.  Concept  of  Robotic  Pipe  Laying  by  Laser  Guidance 


1.5  Basic  Premises  of  Robotic  Excavation  and  Pipe  Laying 

1)  Advanced  sensing  devices  directly  mounted  on  the  excavator  boom  allow  not 
only  the  electronic  detection  of  position  of  the  manipulator  and  forces  within  the 
manipulator  and  the  hydraulic  power  system  but  also  the  detection  of  metals  (e.g., 
pipes)  and  possibly  other  buried  material  which  should  not  be  damaged.  The 
sensory  system  has  the  flexibility  to  be  interfaced  with  on-board  equipment 
monitoring  devices  (e.g.,  oil  temperature)  as  well  as  CAD  software  providing  data 
related  to  the  trenching  and/or  pipe-laying  operation  (e.g.,  depth  of  trench)  and 
electronic  maps  of  underground  utilities.  These  technologies  would  eliminate  the 
need  for  crew  members  climbing  into  the  hazardous  trench.  Also,  many  costly 
accidents  due  to  the  damage  of  buried  utilities  will  be  avoided. 

2)  Shared  control  between  the  operator  and  an  intelligent  and  sensor  based 
controller.  Feedback  of  bucket  position  and  encountered  obstacles  to  the  operator  in 
3-D.  Presentation  of  virtual  views  of  obstacles  or  positions  which  are  presently  not 
available  to  the  operator.  This  operator  augmentation  technology  will  make  the 
entire  operation  more  efficient. 

3)  Real  time  position  control  through  feedback  from  local  positioning  systems  as 
well  as  laser  guidance  systems  such  as  traditional  beam  and  rotating  plane  lasers. 
This  would  eliminate  the  need  for  staking  out  the  trench  and  possible  delay  of 
operations  because  of  lost  or  dislocated  stakes.  The  operator  of  the  equipment 
would  be  supported  by  visual  targets  on  a screen  and  actual  positions  for  either 
robotic  or  manual  control  of  trenching  and  pipe-laying  operations.  The  control 
system  also  supports  the  automatic  generation  of  as-built  data  to  be  directly  fed  to  an 
electronic  utility  map. 
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4)  Support  of  smart  end-effector  attachments  which  are  able  to  connect 
themselves  to  the  excavator  boom/bucket  without  the  need  for  detaching  the 
bucket.  Quick  disconnects  allow  for  easy  hook-up  of  both  electronic  cables  and 
hydraulic  power.  The  smart  end-effector  for  the  laying  of  pipes  includes  a set  of 
specialized  sensors,  such  as  a laser  receivers,  that  are  directly  integratable  with  the 
overall  sensory  system.  The  pipe-manipulator-attachment  is  capable  of 
automatically  grabbing,  positioning,  and  joining  the  pipes  for  storm  drain  or  sewer 
lines  (using  o-ring  joints)  supported  by  a remote  positioning  system  (e.g.,  laser). 


1.  6 The  State-of-the-Art  Experimental  Research  Facility  at  NCSU 

A prototype  robotic  backhoe  excavator  has  been  built  within  the  Construction 
Automation  and  Robotics  Laboratory  (CARL)  of  North  Carolina  State  University 
(for  complete  description,  see  Appendix  A).  This  robotic  manipulator  is  designed  to 
serve  as  a platform  for  many  different  applications  such  as  soil  excavation,  pipe 
laying,  rock  breaking,  etc.  Figure  3 shows  schematically  the  experimental  robotic 
excavator.  Many  different  end  effectors  (i.e.,  bucket,  gripper)  can  be  mounted  at  the 
end  of  the  arm  to  reassemble  this  device  into  different  experimental  configurations. 
In  this  figure,  the  manipulator  is  equipped  with  a bucket  to  operate  as  a computer 
controlled  backhoe  excavator.  The  two  joy  stick  controllers  also  allow  shared, 
traded,  and  distributed  controls.  This  robotic  manipulator  has  a reach  of  6 meters 
(16  feet)  with  heavy  lifting  capacity.  It  is  driven  by  one  hydraulic  motor  and  three 
hydraulic  actuators  (cylinders)  which  provide  a total  of  four-degree-of-freedom 
(DOF).  Each  hydraulic  line  is  equipped  with  an  electronic  pressure  transducer.  Most 
recently,  a vision  system  has  been  added  to  allow  remote  operation  in  shared,  traded 
or  distributed  mode. 

One  386  personal  computer  is  used  to  control  the  entir  robotic  manipulator  and 
to  collect  data  from  the  sensors  in  real  time.  Two  A/D  data  collection  boards  and  a 
D/A  control  board  act  as  interfaces  with  the  electric  transducers  and  hydraulic 
actuators.  Several  sensors  have  been  mounted  for  different  tasks.  The  joint 
encoders  are  used  to  measure  the  individual  joint  displacements  during  excavation. 
Accelerometers  mounted  on  the  boom  and  the  stick  provide  data  about  the 
accelerations  as  well  as  the  inclinations.  In  order  to  detect  an  underground  obstacle 
such  as  a rock,  one  load  cell  has  been  mounted  between  the  rod  and  the  clevis  of  the 
actuator  driving  the  bucket.  To  detect  metal  pipelines,  one  metal  detector  search  coil 
has  been  installed  and  tested  on  the  excavator  arm  (see  Appendix  B).  Its  present 
capabilities  allow  also  contour  mapping  of  obstacles  as  a basis  for  deciding  about 
effective  methods  of  removal  (if  appropriate). 
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Figure  3.  Experimental  Facility  For  Robotic  Backhoe  Excavation 


2.  STATEMENT  OF  RESEARCH  OBJECTIVES 

Researchers  in  CARL  have  worked  for  some  time  on  studying  and  developing  a 
robotic  backhoe  excavator  (see  Appendix  A).  The  goal  is  to  create  a system  which 
will  be  more  accurate  and  efficient  than  traditional  excavators  by  retrofitting  already 
employed  backhoes  with  automated  controls.  While  this  work  is  still  ongoing,  the 
laboratory  is  interested  in  developing  new  technologies  and  finding  new 
applications  in  the  area  of  construction  automation.  The  proposed  research  project 
is  organized  into  two  phases:  A)  Laboratory  Experimentation,  and  B)  Field  Testing. 
The  two  phases  are  further  broken  down  into  various  workpackages. 

Phase  A)  Laboratory  Experimentation: 

Workpackage  A.l:  Mechanical  Hardware 
Workpackage  A.2:  Sensor  Systems 
Workpackage  A.3:  Controls 

Workpackage  A.4:  System  Integration  and  Operator  Interface 

Phase  B)  Field  Testing: 

Workpackage  B.l:  Field-Hardening  of  Prototype  Systems 
Workpackage  B.2:  Field  Testing  in  Training  Ground 
Workpackage  B.3:  On-Site  Field  Demonstrations 
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The  overall  objective  of  the  research  project  is  to  develop  a prototype  system  that 
will  be  able  to  demonstrate  the  technology  in  the  field  and  provide  a basis  for 
designing  a marketable  system  in  the  future.  Several  intermediate  objectives  have 
been  identified. 

Objective  1.  Proof  of  the  technical  feasibility  to  robotically  excavate  trenches  based 
on  information  from  a CAD  database.  It  includes  avoidance  of  non-removable 
obstacles,  such  as  buried  non-metallic  pipes. 

Objective  2.  Testing  and  verification  of  the  already  developed  arm-mounted 
metal  detection  system  under  field  condition. 

Objective  3.  Assessment  of  the  speed  and  accuracy  of  laser  guided  robotic 
excavation  of  trenches  in  the  field. 

Objective  4.  Assessment  of  the  speed  and  accuracy  of  laser  guided  pipe-laying  in 
the  field. 

Objective  5.  Development  and  evaluation  of  human-machine  interfaces  to 
provide  an  effective  operator  interface. 


3.  RESEARCH  PLAN 
A.l:  Mechanical  Hardware 

Design  and  development  of  necessary  hardware  attachments  to  upgrade  the 
existing  experimental  research  facility.  Because  of  the  advanced  state  of  the  existing 
system  this  effort  will  concentrate  on  the  pipe-laying  hardware  and  its  hydraulic 
manipulation  system.  The  mechanism  will  be  able  to  handle  a variety  of  different 
pipe  sizes. 

A.2:  Sensor  Systems 

Several  important  sensing  capabilities  have  already  been  tested.  They  include:  a) 
position  sensing  for  excavator  within  the  construction  site,  b)  position  sensing  for 
bucket,  c)  force  sensing  systems  for  excavation,  d)  force  sensing  systems  for  pipe- 
laying, e)  arm-mounted  metal  detection  system,  and  d)  position  and  force  sensing 
for  joining  pipes.  The  main  addition  to  the  already  existing  sensing  system  are  the 
spatial  positioning  system  for  guidance  and  for  direct  integration  with  a CAD 
representation  of  trench  and  pipe.  Such  a system  will  also  allow  the  establishment 
of  as-built  infomation  automatically.  Several  spatial  positioning  systems  are  on  the 
market  (e.g.,  CAPSY  by  Spectra-Physics)  while  new  ones  are  under  development.  It 
is  proposed  to  purchase  one  of  those  systems  to  be  integrated  with  the  excavator 
hardware  shown  in  Fig.  3. 
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A separate  effort  will  concentrate  on  the  full  development  and  field  testing  of  the 
metal  detection  technology.  With  the  help  of  a company  that  manufactures  metal 
detectors  for  the  construction  industry,  a prototype  metal  detector  search  coil  has 
already  been  mounted  on  the  excavator  arm.  Figure  4 presents  the  schematic  of  the 
presently  deployed  system. 


Figure  4.  Schematic  of  the  Arm-Mounted  Metal  Detection  Hardware 

As  shown,  the  active  search  coil  assembly,  which  has  not  yet  been  optimized,  is 
directly  mounted  onto  the  excavator  arm.  Different  from  passive  metal  detection 
systems,  this  method  does  not  require  any  additional  hardware.  Preliminary  tests 
have  demonstrated  that  the  system,  that  is  directly  integrated  with  the  computer,  is 
able  to  provide  sufficient  sensory  data  to  detect  metal  pipes.  Extensive 
experimentation  and  tests  will  be  conducted  to  verify  the  accuracy  and  effectiveness 
of  the  system.  The  work  would  include  experiments  in  different  soil  conditions  (e. 
g.,  wet)  and  with  different  metal  objects.  The  final  product  would  be  a prototype 
device  that  could  be  mounted  on  traditional  excavators,  would  warn  operators 
about  the  existence  of  any  buried  metallic  obstacle  and  would  indicate  the  position 
of  the  obstacle  in  front  of  the  excavator  bucket.  If  successful,  such  a system  could  be 
directly  implemented  in  the  field  as  an  attachment. 

Pipe  laying  requires  accurate  measurements  and  fine  adjustments.  It  is 
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envisioned  that  through  the  use  of  laser  guidance  and  electric  controls,  pipes  of 
different  sizes  can  be  positioned  efficiently  and  accurately.  Figure  5 depicts  a 
laboratory  experiment  for  pipe  laying  using  laser  guidance. 


Computer 


Figure  5.  Experimental  Laboratory  Set-Up  For  Laser  Integrated  Control 

The  availability  of  laser  guidance  together  with  the  mechanical  means  to 
manipulate  and  connect  drainage  pipes  with  o-ring  joints  will  eliminate  the  need 
of  workers  in  the  trench.  At  the  same  time,  laser  systems  already  provide  very 
precise  guidance  needed  to  put  the  pipes  in  place  accurately.  Total  integration  of  this 
technology  with  the  existing  robotic  manipulator  in  the  laboratory  and  in  the  field 
will  be  the  focus  of  this  sensor  system. 

A»3;  Controls 


This  work  package  will  focus  on  the  development  of  control  models  and  control 
laws  that  provide  the  necessary  algorithms  for  executing  the  different  tasks  and 
subtasks  within  a fully  integrated  framework.  For  that  purpose,  an  extensive  list  of 
experiments  will  be  conducted  as  a basis  for  establishing  a basis  for  adaptively 
controlling  the  manipulator.  Due  to  the  unstructuctured  environment  of  soil 
excavation,  obstacle  recognition,  pipe  handling,  etc.,  different  control  methods  have 
to  be  tested  and  verified  (e.g.,  neural  networks).  In  addition,  virtual  reality  will  be 
used  in  order  to  enhance  the  sensory  based  information  of  the  environment  (e.g., 
approximate  location  of  a pipe).  The  main  task  is  to  design  a control  software  which 
has  the  capability  of  letting  the  manipulator  follow  a planned  trajectory  and  adjust 
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its  path  through  sensing  feedback.  Because  of  existing  underground  obstacles,  such 
as  metal  pipes  and  rocks,  the  control  system  should  also  be  able  to  develop 
methodologies  for  robotic  obstacle  detection,  obstacle  surface  mapping,  and 
automatic  handling  for  underground  objects.  Preliminary  research  work  has 
already  been  conducted  in  CARL. 

Similarly,  control  methods  for  gripping,  manipulation,  positioning,  and  joining 
of  large  concrete  pipes  with  o-ring  joints  will  be  developed.  Effective  use  of  sensor 
systems  (e.g.,  force)  and  the  human  capabilities  of  an  operator  will  be  critical. 

A.4;  System  Integration  and  Operator  Interface 

CAD  databases,  sensory  systems,  and  control  models  for  the  different  tasks, 
organized  in  modules,  will  be  integrated  into  an  overall  control  structure.  Because 
of  the  importance  of  creating  user-friendly  human-machine  interfaces,  special 
attention  will  be  given  to  develop  operator  consoles  for  on-board  and  remote 
controls.  Many  important  lessons  will  be  learned  from  studying  and  testing 
different  interaction  schemes. 

B:  Field  Testing 

The  work  in  phase  A will  be  followed  by  field  tests  with  large  size  excavator 
equipment.  For  that  purpose,  the  developed  technology  will  be  modified  and 
"hardened"  to  survive  the  construction  environment.  Subsequently,  the  hardware  and 
software  will  be  mounted  on  a rented  (possibly  loaned)  excavator  to  be  tested  on  a 
training  ground.  In  a later  phase,  the  robotic  system  will  be  field  tested  in  cooperation 
with  the  North  Carolina  Department  of  Transportation  and  contractors  affiliated  with 
the  Construction  Automation  and  Robotics  Laboratory. 
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4.  PROJECT  SCHEDULE 


Year  5 

Year  1 

Year  2 

Year  3 

Year  4 

8 10  12 

2 4 6 8 10  12 

2 4 6 8 10  12 

2 4 6 8 10  12 

2 4 6 8 10  12 

2 4 6 

Mech.  Hardware 

xxxxxxxxxxxxxx 

Sensor  Systems 

xxxxxxx 

(XXXXXXXXXXXXXX 

Controls 

xxxxxx 

cxxx  xxxxxx  xxxxxx 

xxxxxx 

System  Integration 
and  Operator 
Interface 

XXXXXXXXXXX XX 

Field-Hardening 

xxxxxxx 

Field  Testing  in 
Training  Ground 

xxxxxx 

On-Site  Field 
xxxxxxx 

Demonstrations 

Possible  start-date:  July  1994. 
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5.  RESOURCES  AND  BUDGET 


Table  1:  Resource  Allocation  and  Sources  of  Support 


Resources 

Phase  A 
Laboratory  Exp. 

(3.5  Years) 

Phase  B 

Field  Testing 

(1.5  Years) 

Total 

2 Principal  Investigators 

100,000 

$ 

45,000 

$ 

145,000 

Research  Associate 

$ 

90,000 

$ 

40,000 

$ 

130,000 

Graduate  Res.  Assistants 

$ 

150,000 

$ 

80,000 

$ 

230,000 

Fringe  Benefits 

$ 

40,000 

$ 

17,000 

$ 

57,000 

Travel 

$ 

9,000 

$ 

7,000 

$ 

16,000 

Tuition  Remission 

$ 

35,000 

$ 

15,000 

$ 

50,000 

Electronic  Equipment 

$ 

8,000 

$ 

20,000 

$ 

28,000 

Computer  Hardware 

$ 

10,000 

$ 

10,000 

Vision  System 

$ 

8,000 

$ 

8,000 

Pipe  Laying  Endeffector  Att. 

$ 

10,000 

$ 

2,000 

$ 

12,000 

Small  Equipment  Supplies 

$ 

5,000 

$ 

5,000 

$ 

10,000 

Excavator  Rental 

$ 

9,000 

$ 

9,000 

Equipm.  Transportation 

$ 

5,000 

$ 

5,000 

Spatial  Positioning  System 

. $ 

20,000 

$ 

20,000 

Laserplane  Grade  Control 

$ 

2,000 

$ 

2,000 

$ 

4,000 

TOTAL 

T 

469,000 

$ 

265,000 

$ 734,000 
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Automating  Bridge  Inspection  and  Maintenance 
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Bridge  inspection  and  maintenance  offers  a significant  opportunity  to  reduce  costs 
through  automation,  while  achieving  enhanced  environmental  control.  The 
current  situation  with  respect  to  bridge  maintenance  is  in  fact  so  severe  that  many 
states  cannot  afford  to  attend  to  all  of  the  bridges  in  need  of  repair,  and  are 
focusing  only  on  the  most  severe  cases.  In  consequence,  any  potential  cost 
savings  could  be  expected  to  contribute  directly  to  public  safety  by  extending 
available  funds  to  repair  increased  numbers  of  bridges.  A 1991  report  from  the 
Federal  Highway  Administration  (FHWA)  notes  that  40%  of  the  bridges  on  the 
federal  aid  system  were  built  in  the  1950s  and  60s  with  minimum  cover  for 
reinforcing  steel  and  no  corrosion  protection. 

More  than  a quarter  million  bridges  are  eligible  for  federal  aid.  Of  these,  FHWA 
estimates  that  75,000  need  work.  In  addition  to  Federal  funding,  state  and  local 
governments  spend  between  $500  million  and  $1  billion  a year  on  1,500  to  2,500 
bridges  that  receive  no  federal  money.  All  levels  of  government  combined  will 
spend  more  than  $4  billion  a year  on  bridge  maintenance. 

Increased  environmental  regulations  have  resulted  in  delayed  maintenance, 
particularly  in  the  stripping  and  repainting  of  highway  overpass  structures. 
Increased  process  restrictions  and  media  waste  disposal  costs  have  been  the 
primary  cause  of  delays  and  additional  costs.  Environmental  regulations  now 
require  the  isolation  of  blast  areas  and  have  significantly  increased  waste  disposal 
costs,  especially  for  paints  containing  hazardous  wastes  such  as  lead.  In  past 
years,  a contractor  could  remove  an  overpass's  coating  system  through  open  air 
blasting.  However  open  air  blasting  produces  airborne  contaminants  which  are 
harmful  to  personnel,  equipment,  and  the  environment.  Workers  must  now 
operate  equipment  in  limited  visibility  containment  areas  while  wearing 
cumbersome  protective  suits.  Containment  areas  are  constructed  with  scaffolding 
and  are  enclosed  by  tarps. 

The  erection  of  the  containment  area  is  a slow  and  expensive  process  which 
adversely  affects  productivity.  A North  Carolina  Department  of  Transportation 
case  study  of  the  stripping  and  primer  application  of  two  bridges  in  1991,  found 


that  50%  of  the  time  in  a typical  work  day  was  spent  moving  and  sealing  the 
containment  enclosure  and  cleaning  and  vacuuming  debris. 

The  automation  projects  proposed  in  this  document  would  address  these  needs 
with  a series  of  near-term  and  longer-term  developments.  Automated  systems 
would  remotely  inspect  bridge  paint  condition  and  the  physical  state  of  bridge 
members,  erect  shrouds,  recover  stripping  media  and  paint  with  increased 
efficiency,  and  in  general,  reduce  the  manpower  and  time  required  to  strip  and 
paint  an  overpass.  A simple  teleoperated  system  would  provide  the  following  key 
benefits: 

• Increase  speed  and  efficiency  of  inspection  operations. 

• Reduce  time  required  for  placement  of  shrouding. 

• Reduce  time  required  to  strip  paint. 

• Reduce  exposure  of  workers  to  harmful  materials. 

• Reduce  environmental  impact  of  paint  stripping. 

• Reduce  hazardous  waste  through  stripping  media  recycling. 

• Reduce  hazardous  waste  through  more  efficient  paint  application. 

• Improve  quality  of  painting  and  decrease  time  required  for  application. 

• Automate  generation  of  maintenance  and  inspection  databases. 

• Reduce  manpower  requirements. 

For  typical  highway  overpass  bridges,  it  appears  feasible  to  reduce  the  costs  of 
stripping  and  painting  by  one  quarter  to  one  third,  while  also  providing 
environmental  advantages. 
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1 Approach 

Technology  is  available,  or  nearly  available,  today  for  automating  the  remote 
collection  of  imaging  and  other  sensor  data  to  assess  bridge  and  paint  condition. 
Similarly,  automated  tooling  for  stripping  and  applying  paint  is  available.  The 
principal  problems  to  be  addressed  are  the  automated  placement  and  manipulation 
of  such  sensors  and  tooling,  the  automated  placement  of  shrouding,  and  ultimately 
the  automated  interpretation  and  logging  of  data  and  control  of  operations. 

An  initial,  near-term,  step  is  to  develop  improved  alternatives  to  “Snoopers”  for 
positioning  and  manipulating  automated  bridge  inspection  and  maintenance 
operations.  Advanced  control  techniques,  already  developed  for  industrial 
robots,  could  be  used  to  provide  coordinated  motion  for  devices  adapted  from 
existing  machines,  such  as  aerial  lifts  or  cranes.  Two  such  concepts  are  presented 
in  this  proposal  for  the  development  of  a prototype  Robotic  Highway  Overpass 
Stripping  (RHOS)  System.  These  devices  could  begin  to  achieve  a number  of  real 
benefits,  even  with  initial  simple  teleoperated  modes  of  operation. 

To  the  extent  possible,  NIST  intends  to  borrow  from  technology  being  developed 
for  advanced  paint  removal  processes  in  other  industries  and  apply  it  in  the 
development  of  a RHOS  system.  NIST  recently  completed  an  investigation  of 
current  DoD  and  civilian  efforts  aimed  at  automating  paint  stripping  of  both 
aircraft  and  Navy  vessels. 

An  effort  is  underway  within  the  Armed  Services  Joint  Depot  Maintenance 
Program  called  the  Joint  Paint  Removal  Study.  Under  this  effort,  five  lead  depots 
have  been  designated  for  projects  aimed  at  developing  alternative  stripping 
processes.  As  is  the  case  with  bridge  stripping,  the  DoD  requirements  are  driven 
by  environmental  regulations.  The  DoD  efforts  will  encompass  the  development, 
testing,  and  in  some  cases,  automation  of  alternative  stripping  processes. 

NIST  anticipates  involvement  in  the  DoD  program  through  the  application  of  the 
NIST  Real-time  Control  System  (RCS)  technology  to  the  development  of  a 
coordinated  motion  boomlift.  This  telerobotic  aerial  lift  will  then  be  integrated 
with  paint  blast,  media  recovery  and  recycling  equipment  now  being  developed. 
The  project  will  result  in  the  development  of  a prototype  telerobotic  machine  for 
ship  hull  paint  stripping  in  Navy  ship  yards.  The  goal  is  to  use  as  much  standard, 
or  off-the-shelf  components  as  possible  to  produce  a productive  and  reliable 
system  at  a low  cost. 
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Much  of  this  technology  would  also  be  applicable  to  bridge  stripping.  Although 
the  configuration  of  lift  equipment  chosen  for  bridge  maintenance  may  differ 
from  that  used  for  ship  hull  stripping,  much  of  the  control  software  and 
hardware  would  be  applicable  to  both.  For  example,  a mobile  platform- mounted 
lift  is  anticipated  for  use  in  ship  hull  stripping,  whereas,  the  use  of  a truck- 
mounted  articulated  lift  might  be  a likely  candidate  for  bridge  maintenance. 
However,  this  would  require  only  slight  variations  for  the  implementation  of  an 
RCS.  In  addition  much  of  the  technology  that  is  now  being  developed  for 
processing  equipment  under  DoD  projects  would  also  likely  be  applicable  to  the 
maintenance  of  bridges. 

Initial  efforts  for  a bridge  maintenance  automation  project  would  be  focused 
toward  the  development  of  a coordinated  motion  aerial  lift  or  truck  crane.  A 
teleoperated  prototype  machine  would  be  produced  that  is  controlled  via  a 
joystick  using  intuitive  inputs  on  the  part  of  the  operator.  The  operator  would 
simply  move  the  joystick  in  the  direction  in  which  he  desired  the  equipment  (e.g. 
stripping  or  painting  head)  to  move. 

In  the  longer  term,  wholly-new  forms  of  robotic  devices  could  be  designed  for 
delivery  of  many  inspection  and  maintenance  services.  Such  machines  would  be 
programmed  from  databases  containing  descriptions  of  each  bridge  along  with  its 
associated  maintenance  requirements.  These  databases  would  constitute  the  input 
for  planning  and  scheduling  bridge  repair,  and  for  programming  the  robots  to 
perform  various  functions  such  as  inspection,  paint  stripping,  and  repainting. 

Advanced  robotic  devices  working  from  such  databases  could  be  developed  to 
crawl  along  over  and  under  bridges,  automatically  performing  routine  inspection, 
paint  stripping,  and  repainting  chores.  While  large  bridges  might  have  custom- 
designed,  dedicated  robots,  robots  might  also  be  specially  designed  for  whole 
classes  of  smaller  bridges.  Such  robots  would  be  transported  to  bridge  sites  and 
deployed  to  crawl  along  a bridge  and  associated  support  structures  and  monitor  a 
number  of  parameters  to  monitor  the  health  of  the  bridge.  Physical 
characteristics  that  could  be  monitored  include;  macro-scale  deformations,  paint 
condition  and  corrosion.  The  same  robots,  while  still  in  place  and  fitted  with 
different  apparatus,  would  then  perform  stripping,  painting,  and  perhaps  minor 
structural  repair  as  indicated  by  the  inspection. 
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2 Near-Term  Technical  Objectives 

Bridge  inspection  and  maintenance  is  a major  activity  involving  numerous 
bridges  and  overpasses  ranging  widely  in  both  size  and  complexity.  A variety  of 
automation  opportunities  appear  as  alternatives  to  the  current  use  of  “Snoopers” 
and  scaffolding  for  manual  operations. 

The  NIST  Robot  Systems  Division  has  been  developing  flexible,  intelligent 
control  systems  for  over  a decade.  The  foundation  of  these  control  systems  lies  in 
a control  architecture  which  establishes  a skeletal  framework  for  building  flexible 
controls  for  complex  systems.  This  sensor  based,  NIST  Real-time  Control 
System  (RCS)  architecture  has  been  implemented  in  numerous  robot,  vehicle,  and 
factory  control  applications.  These  systems  include  serial  link  robots, 
cooperative  robots,  coordinated  robots,  factory  automation,  automatic  guided 
vehicles,  autonomous  underwater  vehicles,  teleoperated  robots,  teleoperated 
vehicles,  and  parallel  link  manipulators. 

The  Robotic  Highway  Overpass  Stripping  (RHOS)  system  could  overcome  many 
of  the  environmental  barriers  confronting  highway  overpass  maintenance.  RHOS 
would  be  an  integrated  set  of  manipulators,  stripping  devices,  and  containment 
shrouds  under  the  coordinated  control  of  an  RCS  system.  RHOS  would  provide 
rapid  set  up  and  use,  complete  recovery  of  the  blast  media  and  paint  debris, 
maximum  recycling  of  the  stripping  media,  and  the  exclusion  of  human  workers 
from  the  blasting  environment. 

An  overpass  stripping  operation  using  RHOS  would  consist  of  a robotized  aerial 
lift  capable  of  manipulating  a blasting  nozzle  and  a containment  shroud  along  the 
structural  members  of  highway  overpasses.  The  containment  shroud  would 
transport  media  waste  to  a recovery  system.  The  recovered  blasting  media 
would,  to  the  extent  possible,  be  recycled. 

The  goal  of  this  near-term  demonstration-effort  would  be  to  develop  and 
demonstrate  a robotically  manipulated  paint  stripping  system.  This  RHOS  system 
would  consist  of  a robotized  aerial  lift  capable  of  manipulating  blasting  nozzles 
and  containment  shroud  along  structural  members  of  highway  overpasses.  RHOS 
would  initially  be  largely  teleoperated  and  would  be  designed  for  ease  of 
operation  by  current  overpass  maintenance  contractors.  The  operator  interface 
would  be  designed  to  minimize  the  operator's  programming  tasks  and  support  on- 
site diagnostics.  The  RHOS  system  would  include  an  integral  shroud  subsystem, 
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expected  to  recover  a minimum  of  ninety  five  percent  of  the  spent  media  and  chip 
debris.  The  shroud  subsystem  will  also  transfer  the  debris  to  a recycling 
subsystem.  RHOS  would  clean  overpass  metal  structures  to  near  white  metal  by 
media  blasting,  and  produce  an  appropriate  surface  profile  for  new  coatings.  In 
its  initial  form,  the  RHOS  system  should  be  able  to  access  those  surfaces  routinely 
addressed  by  manual  blasting. 


2.1  Technical  Approach 

Development  of  the  RHOS  system  will  require  two  major  efforts.  One  effort  will 
focus  on  the  development  of  a computer  controlled  aerial  lift  to  maneuver  in  and 
around  highway  overpass  structures.  The  second  effort  will  focus  on  the 
development  of  the  processing  equipment  which  includes  all  blasting, 
containment,  and  recovery  subsystems. 

Initial  efforts  for  the  bridge  maintenance  automation  project  would  be  directeded 
toward  the  development  of  a coordinated  motion  aerial  lift  or  truck  crane.  A 
teleoperated  prototype  machine  would  be  produced  that  is  controlled  via  a 
joystick  using  very  intuitive  inputs  on  the  part  of  the  operator.  The  operator 
would  simply  move  the  joystick  in  the  direction  in  which  he  desired  the 
equipment  (e.g.  stripping  or  painting  head)  to  move.  Currently  available  aerial 
lifts  are  not  capable  of  making  a smooth  coordinated  move,  such  as  moving  in  a 
straight  line  along  the  surface  of  a bridge  girder.  Each  joint  must  be  controlled 
independently  by  the  operator. 

NIST  has  held  discussions  with  major  aerial  lift  manufacturers  about  establishing 
a Cooperative  Research  and  Development  Agreement  (CRADA)  in  which  a 
coordinated  motion  aerial  lift  would  be  jointly  developed  between  NIST  and  one 
or  more  lift  manufacturers.  Following  a rapid  development  period,  this 
prototype  could  be  duplicated  in  low-cost  production  versions  for  use  in  common 
maintenance  operations. 

Subsequent  implementation  of  higher  levels  of  the  RCS  control  hierarchy,  along 
with  sensory  feedback  would  provide  increased  autonomy  of  this  equipment, 
resulting  in  higher  productivity  and  greater  cost  savings.  This  effort  would  also 
provide  a foundation  for  the  automation  of  other  similar  equipment,  providing 
additional  benefits  to  the  FHWA  in  other  operations. 
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These  initial  efforts  would  entail  interface  developments  for  additional 
mechanical  joints  and  command,  status,  and  sensor  communications.  Efforts 
would  also  include  the  development  of  computer,  electronic,  and  sensor 
components  needed  for  robotic  control  with  minimal  operator  interaction.  The 
RHOS  system  controller  will  supervise  both  the  aerial  lift  and  the  processing 
equipment.  NIST  would  develop  the  controller  in  accordance  with  the  NIST  Real- 
time Control  System  architecture  (RCS)  including  an  easy-to-use  operator 
interface,  such  that  the  operator  will  require  neither  specialized  computer  nor 
robotic  training.  The  RCS  hierarchal  structure  would  ensure  the  RHOS  control 
system  remain  readily  adaptable  to  future  expansion. 

Existing  systems  for  acquiring  sensory  data  and  for  maintaining  camera  fixation 
through  coordination  of  camera  motion  with  lift  motion  could  easily  be  adapted 
to  RHOS.  NIST  has  already  developed  rapid  camera  gaze-control  systems 
operating  under  the  RCS. 

Concepts  for  two  potential  containment  systems  are  depicted  in  Figure  1 and 
Figure  2.  The  containment  system  must  permit  manipulation  by  the  robotic 
aerial  lift,  must  not  restrict  the  blasting  of  overpass  structural  members,  and  must 
transfer  the  media  waste  to  the  recovery  subsystem.  The  containment  system 
must  also  conform  to  irregularities  on  the  surface  to  be  stripped  in  order  to 
insure  adequate  containment  of  the  waste.  This  may  entail  multiple  modular 
shroud  components  designed  for  rapid  change  or  a single  conformable  device. 
Deployable  shrouding  might  begin  with  a large  emplacement  within  which  the 
robot  mechanisms  could  act  over  a large  volume,  as  shown  in  Figure  1. 

Subsequently,  new  developments  from  DoD  efforts  might  be  adapted  to  provide 
small,  movable  shrouding  as  shown  in  Figure  2. 


Final  system  integration  will  entail  the  assembly,  testing  and  demonstration  of  the 
complete  RHOS  system.  The  system  integration  phase  of  the  project  would 
conclude  with  field  demonstrations  of  the  RHOS  system,  in  which  various 
overpass  structural  members  would  be  stripped. 
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2.2  Major  Tasks 

Twelve  tasks  are  proposed  for  the  RHOS  project.  This  initial  tasks  (1-6)  could  be 
accomplished  over  the  first  12  months,  with  a demonstration  of  the  retrofit 
coordinated  motion  manipulator,  possibly  performing  an  inspection  operation. 
The  remaining  tasks  (7-12)  would  be  completed  in  24  months.  Actual  bridge 
stripping  would  be  demonstrated  using  a prototype  teleoperated  stripping 
machine.  The  project  tasks  are  briefly  outlined  below. 

1.  Identify  Overpass  Features:  Through  cooperative  efforts  with  the  Federal 
Highway  Administration  and  a State  Department  of  Transportation,  select  a 
representative  set  of  highway  overpasses.  The  effort  will  identify  common 
features  or  families  of  features  from  the  selected  overpasses  which  now  require 
periodic  stripping.  The  RHOS  system  will  then  be  developed  to  process  all  of  the 
identified  features. 

2.  Select  Lift/Crane:  Perform  comparative  analysis  of  available  lifts  and  cranes. 
All  configurations  of  commercially  available  lifts  and  cranes  will  be  evaluated  for 
use  in  this  project.  Several  characteristics  will  be  considered,  including  dexterity, 
cost  and  suitability  to  coordinated  motion/teleoperation  development. 

3.  Select  Blasting  System:  Perform  comparative  analysis  of  available  paint 
removal  methods.  Commercial  equipment  is  currently  available  for  several 
different  stripping  processes.  Additional  processes  and  blast  media  are  also  under 
development  in  DoD  and  private  industry.  These  processes  will  be  evaluated  for 
use  in  bridge  stripping.  To  a great  extent,  this  evaluation  will  be  based  on  results 
obtained  from  previous  efforts,  such  as  the  DoD  Joint  Paint  Removal  Study. 

4.  Develop  Shroud  Subsystem:  The  shroud  subsystem  is  considered  to  be  the 
highest  risk  element  in  this  project.  The  shroud  must  fulfill  the  EPA 
requirements  for  level  3 paint  stripping  by  recovering  the  spent  media  and 
debris,  and  transfering  this  material  to  the  recycling  subsystem.  An  evaluation 
will  be  conducted  of  existing  and  emerging  shroud  and  recovery  system  designs. 
Efforts  will  be  made  to  incorporate  demonstrated  technologies  into  the  design  of 
the  RHOS  shroud  subsystem. 

A stationary-type  shroud  design  would  remain  fixed  to  the  bridge  structure  while 
the  blasting  nozzle  maneuvers  within  (see  Figure  1).  In  this  configuration,  some 
of  the  manipulator  joints  operate  within  the  shroud  while  others  remain  outside. 
The  shroud  will  have  a port  which  will  seal  about  the  manipulator  and  ensure  its 
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freedom  of  motion.  Media  waste  will  exit  the  shroud  through  another  port 
primarily  by  means  of  gravity  and  be  routed  to  the  recycling  system. 

With  a dynamic  shroud  design,  the  shroud  would  be  an  integral  part  of  the  end 
effector  carrying  the  blasting  subsystem  (see  Figure  2).  This  form  of  shroud 
would  enclose  the  blasting  unit  and  move  along  a surface  via  the  robotic  aerial 
lift.  The  dynamic  shroud  system  would  most  likely  consist  of  interchangeable 
modules,  each  designed  for  use  on  different  structural  features.  A vacuum 
system  would  route  the  media  waste  to  the  recycling  subsystem.  A shroud  such  as 
this  will  be  tested  in  a Navy  ship  hull  stripping  demonstration  project.  In 
addition,  a small  company  has  demonstrated  a similar  prototype  recovery  system 
for  ship  hull  stripping. 

5.  Select  Computer  Configuration:  Perform  comparative  analysis  of  computer 
hardware  for  control  system.  Select  system  based  on  availability,  support,  ease  to 
interface,  expandability,  convenience  of  field  maintenance,  and  cost. 

6.  Retrofit  Lift/Crane:  The  commercial  lift  or  crane  chosen  for  the  project  must 
be  at  least  partially  retrofitted  to  accommodate  coordinated  joint  motion  control. 
This  will  include  implementation  of  joint  position  sensors,  either  resolvers  or 
encoders.  It  will  also  require  that  all  joints  be  driven  via  electrohydraulic 
proportional  control  valves  having  sufficient  frequency  response  to  allow  for  the 
necessary  level  of  system  performance.  In  addition,  some  re-routing  or 
installation  of  new  electrical  and  hydraulic  cables  will  also  be  required  and  it  may 
be  beneficial  to  implement  the  control  and  power  lines  on  a commercial  bus 
network. 

7.  Design  Manipulator  Extension:  Standard  aerial  lifts  do  not  have  sufficient 
dexterity  to  access  all  of  the  required  areas  of  an  overpass.  Additional  degrees  of 
freedom  (joints)  will  be  incorporated  at  the  end  of  the  aerial  lift  to  provide  the 
necessary  manipulator  dexterity. 

8.  Develop  User  Interface:  The  operation  of  the  RHOS  system  will  require 
minimal  training  for  existing  overpass  maintenance  personnel.  The  controls  shall 
be  intuitive  with  clear  identification.  The  interface  panel  will  also  be  portable 
such  that  the  operator  will  be  able  to  position  himself  at  the  most  advantageous 
position  to  operate  the  system. 

9.  Develop  System  Controller:  A set  of  routines  will  be  developed  in  the  system 
controller  to  perform  all  the  functions  required  of  the  RHOS  system.  Routines 
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will  be  included  which  provide  for  the  stripping  of  specific  overpass  features  and 
collections  of  features,  and  also  for  performing  diagnostics  and  simple 
maintenance. 

10.  Integrate  Aerial  Lift  and  Extension:  This  integration  consists  of  assembling 
mechanical  components,  establishing  controller  communications,  and  establishing 
the  coordinated  motion. 

11.  Integrate  Stripping  System:  The  stripping  system  shall  be  tested  to  ensure 
that  the  blasting  subsystem  can  remove  common  overpass  paints  and  that  the 
recovery  subsystem  contains  and  recycles  the  media  as  required.  Representative 
values  will  be  derived  from  these  tests  and  used  to  establish  the  system's  operating 
parameters.  These  parameters  include  such  items  as  media  flow  rates,  pressures, 
stand-off  distances,  linear  feed  rates,  and  overall  processing  rates. 

12.  Perform  Field  Tests:  The  final  testing  includes  two  phases.  The  first  phase 
could  be  conducted  at  NIST.  The  RHOS  system  will  be  operated  at  the  extremes 
of  its  design  requirements.  Tests  will  ensure  the  robustness  of  the  controller,  the 
thoroughness  of  the  programming,  and  the  convenience  and  completeness  of  the 
operator  interface. 

The  second  phase  will  consist  of  stripping  paint  from  girders  at  one  or  more 
highway  overpass  sites.  The  sites  will  be  selected  by  an  appropriate  state  highway 
department.  The  site  should  be  scheduled  for  routine  paint  stripping  maintenance 
and  the  regular  contractor's  employees  should  operate  the  RHOS  system. 

3.  Longer-Term  Technology 

Advanced  robotic  devices  will  be  designed  with  the  ability  to  perform  inspection, 
paint  stripping,  and  repainting  operations  by  maneuvering  along  the  I-beams 
under  existing  bridges.  An  example  of  such  a robot  is  illustrated  in  Figure  3. 

This  device  consists  of  two  crawling  manipulators  (arms)  and  a work  platform 
with  a robot  manipulator  and  containment  shroud  (shroud  not  shown  in  the 
figure). 
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3.1  Advanced  Automation 


The  initial  test  version  of  this  device  will  function  under  teleoperation  via  an 
operator  using  a joystick  or  space-ball.  However,  a programmable  system  will 
have  an  control  system  that  will  retrieve  geometry  information  from  a database 
describing  the  particular  bridge  as  well  as  a set  of  actions  for  performing 
prescribed  tasks.  The  bridge  database  will  be  developed  originally  from  an  "as- 
built"  database,  which  in  turn  will  be  updated  each  time  the  bridge  is  repaired  or 
otherwise  modified.  This  database  will  provide  the  input  for  planning  and 
scheduling  bridge  repair,  as  well  as  for  programming  the  robot  to  perform  the 
various  functions  required  for  inspection,  paint  stripping,  and  repainting. 

The  design  envisioned  would  operate  under  the  guidance  of  a hierarchical  control 
system,  such  as  the  NIST  RCS  design.  The  controller  would  retrieve  database 
information  describing  details  of  a particular  bridge,  and  adapt  subsequent 
operations  under  a generic  task  plan  that  would  be  applicable  to  all  bridges  of  the 
that  same  class.  This  would  be  a straightforward  extension  of  current  state-of- 
the-art  industrial  practice.  For  example,  in  the  robotic  welding  of  automobiles, 
new  CAD  designs  are  accessed  to  generate  paths  for  robot  welders.  Since  the 
nature  of  operations  in  the  field  is  inherently  less  structured  and  less  controllable 
than  in  factory  situations,  this  level  of  operation  would  still  require  general 
guidance  by  an  operator,  but  routine,  detailed  motion  planning  in  real-time  would 
be  generated  by  the  task-planning  program. 

In  a later  stage  of  development,  the  device  would  incorporate  sensing  for 
guidance  in  addition  to  the  initial,  specialized  sensing  for  inspection.  This  would 
allow  the  machine  to  adapt  to  errors  in  positioning  and  to  new  situations  not 
contained  in  the  data  base  (e.g.,  bird's  nests,  undocumented  repair  work). 

Sensory-guided  robotic  systems  of  this  sort,  operating  under  general  task  plans 
and  working  from  apriori  databases  exist  today  in  laboratory  settings  and 
experimental  operations.  They  can  successfully  adapt  simple  planned  operations 
such  as  navigation  and  manipulation  to  minor  discrepancies  between  their 
observed  sensory  data  and  expectations  generated  from  a priori  databases. 
Experimental  approaches  to  automatically  updating  the  databases  from  the  robot's 
sensory  input  also  exist.  The  bridge-maintenance  task  presents  an  attractive 
target  for  applying  and  extending  these  experimental  technologies  to  highway 
maintenance  operations  due  to  the  relatively  fixed  nature  of  the  task,  the 
relatively  structured  environment,  and  the  relative  ease  with  which  working 
databases  could  be  produced. 
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4.0  Estimated  Cost 


NIST  has  held  discussions  with  aerial  lift  manufacturers  regarding  the 
development  of  a coordinated  motion  machine.  It  is  anticipated  that  a CRADA 
will  be  established  between  NIST  and  at  least  one  of  the  manufacturers  in  which 
manufacturer  would  contribute  the  use  of  a lift  for  the  duration  of  this  project. 
The  following  is  an  estimate  for  the  cost  of  this  proposed  project,  assuming  the 
manufacturer  contributes  the  lift. 

Phase  I - Develop  Inspection  Capability  (12  months): 


Identify  Overpass  Features 

$ 20,000 

Select  Lift/Crane 

15,000 

Select  Blasting  System 

20,000 

Develop  Shroud  Subsystem 

70,000 

Select  Computer  Configuration 

10,000 

Retrofit  Lift/Crane 

80.000 

$215,000 

Phase  II  - Develop  and  Demonstrate  Paint  Stripping  (24  months): 


7.  Design  Manipulator  Extension 

30,000 

8.  Develop  User  Interface 

35,000 

9.  Develop  System  Controller 

65,000 

10.  Integrate  Aerial  Lift  and  Extension 

30,000 

11.  Integrate  Stripping  System 

35,000 

12.  Perform  Field  Tests 

40,000 

1 3 . Document  Results 

and  Transfer  Technology 

40.000 

$285,000 

Total 

$500,000 
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Figure  1.  Possible  Stationary  Shroud  Configuration 
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Figure  2 Possible  Dynamic  Shroud  Configuration 
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